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Abstract 

The present study piloted a survey-based measure of Opportunity to Learn (OTL) and 
Academic Language Exposure (ALE) in fourth grade science classrooms that sought to 
distinguish teacher practices with ELL (English language learner) and non-ELL students. 
In the survey, participant teachers reported on their instructional practices and the context 
in their science classrooms. A small sub-sample was also observed teaching a lesson in 
their classroom on two occasions. The pilot data were used to investigate basic 
psychometric properties of the survey: specifically (a) the dimensions underlying the 
survey items, in particular whether OTL and ALE are distinct or overlapping features or 
dimensions of science instruction and (b) the match between information reported by 
teachers in the survey, and that collected by classroom observers. Qualitative analyses of 
observation and teacher open ended responses in the survey informed the interpretation 
of the quantitative analysis results and provided useful insights for refining the survey 
instrument to better capture the classroom experiences of ELL students. 



Introduction 

The No Child Left Behind Act of 2001 (NCLB, 2002) makes clear that states, districts, 
schools, and teachers must hold the same high standards for English language learners 
(ELLs) as for all other students, and that they are accountable for assuring that all students, 
including ELL students, meet high expectations. One of the problems that test developers and 
users face in validating the content area assessments for ELL students is to determine the 
extent to which the scores reflect differences in the levels of student achievement, and 
whether they are affected by a myriad of classroom and school environmental factors. 
Among these factors, differences in Opportunity to Learn (OTL) and Academic Language 
Exposure (ALE) may be particularly influential in the case of ELLs. Most states do not 
monitor opportunities to learn systematically for ELL students in content areas such as 
mathematics and science, and still fewer know about their exposure to academic language, 
even though both aspects of the learning environment can influence student performance and 
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thus affect the interpretability and validity of academic assessments (see Baker et al., 1995; 
Boscardin, Aguirre-Munoz, Chinen, Leon, & Shin, 2004; Herman & Abedi, 2004; Stevenson 
& Stigler, 1992). 

The opportunity of ELL students to learn content area material and be to exposed to the 
cognitively demanding language of academic contexts can be seen as part of the larger 
concern for educational equity and access experienced by minority students in the United 
States (e.g., Callahan, 2005; Darling-Hammond, 2007), a particularly important agenda in an 
era of increased accountability and large-scale assessment (Moss, Pullin, Gee, Haertel, & 
Young, 2008). Specifically in the case of linguistic minority students’ access to high-quality 
education, lowered academic expectations and tracking practices used with ELL students in 
response to their lower levels of language proficiency are agued to have deleterious effects 
on their school performance (Callahan, 2005). The K-12 schooling experiences of ELL 
students of course go on to impact their access to higher education opportunities (Gandara, 
2005). Knowing more about ELL students’ opportunity to learn content and receive exposure 
to academic English compared with those of their English-proficient or native-English- 
speaking peers can contribute to the ongoing discussion of how best to increase educational 
access for minority students. 

Background: ELL Student Performance, Opportunity to Learn, and Academic 
Language Exposure 

Multiple recent research efforts have probed the validity of assessments used with ELL 
students, by investigating various technical features including alignment with state standards, 
differential item functioning (Oiler & Damico, 1991), and use of accommodations (e.g., 
Abedi, Hofstetter & Lord, 2004) among others. The broad picture that emerges from these 
studies suggests that much is left to be done to ensure the reliability and validity of the 
available information about ELL students’ academic achievement. 

One issue less commonly explored when investigating test validity for ELL students is 
Opportunity to Learn (OTL). In general, concern with OTL arises from the notion that 
students should be adequately exposed to the contents they are tested on. The precise 
definition on OTL depends directly on the interpretation of the term “adequately”; existing 
definitions range from narrow (i.e. OTL as a set of item-level coverage indicators to ensure 
comparability in international assessments) to broad (i.e. OTL as encompassing the 
educational experiences of students in the school, including curriculum, resources, teacher 
quality, instructional practices, and remediation efforts; see e.g., Abedi, Courtney, Leon, Kao 
& Azzam, 2006; Herman & Abedi, 2004; Tindal & Haladyna, 2002). 



2 




Although a broader definition of OTL may pose major challenges of conceptualization 
and operationalization, it also seems clear that a serious attempt at understanding the 
performance of ELL students in standardized assessments must include a careful look at their 
educational experiences in the school or classroom. The key validity question concerns the 
degree to which ELL students’ test scores reflect real variation in achievement, or may be 
explained in terms of systematic differences in the opportunities to learn they are offered. 
Lor example, opportunity to learn the expected content material may be reduced if ELL 
students are taught at a slower pace, miss content classes for language instruction (i.e., attend 
ESL pull-out classes, or are simply not taught to the same depth of knowledge as native 
English-speaking students [Boscardin, et al., 2004; Lrancis, Lesaux, & August, 2006; 
Lrancis, Rivera, Lesaux, Kieffer, & Rivera, 2006; Gutierrez, & Jaramillo, 2006; Herman, 
Klein, & Abedi, 2000]). 

Cruz (2004) provides an exhaustive list of the different ways in which a highly complex 
command of the English language is demanded of students in order to carry out cognitive 
tasks in school: 

We want them to understand lectures and participate in academic conversations. We want 
them to comprehend challenging texts, make informed decisions based on information 
they have read, form rational opinions, and offer focused interpretations. We expect them 
to write with clarity, conviction, color, and sophisticated thought. In short we want them 
to express themselves intelligently, articulately, and thoughtfully (p. 14). 

Thus, student performance may be impacted when they lack exposure to these 
academic uses of English. The low exposure may be attributed to the inaccessibility of the 
language used to teach content, the lack of explicit teaching of academic vocabulary and 
grammatical structures specific to the content areas, and unfamiliarity with the diverse range 
of language functions such as descriptions, explanation, summarizations, etc., used in 
academic settings (e.g., Bailey, Butler, Stevens & Lord, 2007; Scarcella, 2003; 
Schleppegrell, 2004). 

Traditional classroom measures based on teacher reports of content coverage may not 
provide sufficient information about OTL in the case of ELL students. District and school 
programming ELL students receive in order to become sufficiently proficient in English 
varies widely and may greatly impact the degree of student access to content. Lor example, 
they may receive near exclusive instructional focus on English language development, or 
instead content-based ELD instruction, or even content instruction in a bilingual, native 
language, or English-only context. Moreover, they may receive instruction from a teacher 
who currently has little or no training in the teaching of ELL students (NCELA, 2008). 
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Because of their limited mastery of the language, ELL students may not be exposed to 
academic language in English to the same extent as other students in the classroom, and thus 
may not be as familiar with content- specific vocabulary, academic grammatical structures 
and academic language functions. A focus of the current study is investigating whether ALE 
is a separate construct, or can be seen as a particular kind of OTL.^ These constructs are 
discussed in further detail below. 

Research Questions and Methodology 

The main goal of this study was to develop and pilot a survey-based instrument to 
measure Opportunity to Learn and Academic Language Exposure in science classrooms, 
specifically as it relates to the educational experiences of ELL students. With the data 
collected, we additionally set out to investigate whether OTL and ALE are distinct features 
of teacher practice in science classrooms, or can be thought of as part of a broader dimension 
of instruction. We address the following specific research questions: 

1 . Are there differences in the levels of OTL and ALE teachers report offering to ELL 
and non-ELL students in science classrooms? 

2. What OTL and factors (dimensions) underlie the survey items in each section 
of the survey? Do OTL and ALE factors identify distinct features of classroom 
practice? 

3. Is there correspondence between OTL and ALE as reported by teachers in the 
survey, and information obtained through direct classroom observations? 

4. Do participant teachers see the survey as providing a complete picture of OTL/ ALE 
for ELL students in their classroom? What insights for improving the survey can be 
extracted from survey and observation data, and teacher reflections and comments? 

To address these research questions we adopted a mixed-methods approach; we carried 
out statistical analyses to investigate the psychometric properties of the teacher survey (i.e. 
item variance, correspondence between surveys and observations, and interrelationship 
among different survey items and sections). We complemented these quantitative analyses 
with qualitative case studies of five respondents teaching science in their classrooms; the 
case studies provided rich contextual data to inform our interpretations of the quantitative 
results. The sample, instrumentation, and analytic methods are discussed in the following 
sections. 



Student achievement is not only influenced by exposure to OTL and ALE in the formal school setting, but 
also at home or in out-of-school learning contexts. These are outside the scope of the current study, but are 
identified here as an important area of future research. 
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Sample 

We recruited 4th-grade science teachers in California and Colorado to participate in the 
survey pilot in fall 2007 and spring 2008 through announcements in online forums, 
professional development networks, and directly through district’s science offices. Two 
versions of the survey were created to reflect the standards for 4th grade science in the two 
states. Of the 97 teachers who initially volunteered to participate in the study and were 
mailed surveys, we received 53 surveys back, resulting in a response rate of 55%. Participant 
teachers received a small stipend ($30) in gratitude for their assistance with the project. 

In addition to completing the OTL survey, five teachers in three schools in California 
agreed to allow two observers into their classrooms for in-depth observation for 
approximately two one-hour periods of science instruction. The five classrooms cover the 
range of proportions of ELL students that teachers encounter in the elementary grades, from 
only a handful (7%) to ELL-only (100%). These teachers received a stipend of $125 for their 
participation. 

Constructs 

Opportunity to learn. Lor the purpose of this study OTL was operationalized to 
encompass a range of factors that determine the kinds of educational opportunities students 
are exposed to in the classroom. We adapted the four-dimension OTL model first proposed 
by Stevens (1993) and later used by other researchers (e.g. Wang, 1998; Stevens, Wiltz, & 
Bailey, 1998). The four dimensions include content coverage, content exposure, content 
emphasis, and quality of instructional delivery. Content coverage refers to the extent of 
coverage of core content curriculum whereas content exposure reflects the amount of time 
allocated for and devoted to instruction. Content emphasis, captures the areas or skills are 
treated as a major focus in teaching. Linally, quality of instructional delivery refers to 
classroom practices and instructional strategies employed by the teachers in delivering the 
content (Boscardin et ah, 2004; Wang, 1998). Items that capture these four dimensions were 
included in the two OTL sections of the survey: Instructional Activities and Configurations 
and Instructional Emphases. 

We complemented these four dimensions with three ELL-specific facets of OTL 
identified by previous research at the National Center for Research on Evaluation, Standards 
and Student Testing (CRESST; e.g., Boscardin et ah, 2004). Specifically, the ELL 
Instructional Strategies section of the survey included second language acquisition 
strategies, ELL process strategies, and instruction delivery formats. Second language 
acquisition strategies are used to provide students with tools for either making sense of the 
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linguistic input they receive, or producing linguistic output that meets expectations for 
academic discourse (Boscardin et al., 2004). ELL process strategies are generally used to 
minimize the amount of linguistic input in instruction; examples include language 
scaffolding techniques such as use of graphics and students’ primary language, and the 
deliberate use of extra wait time for students’ responses. Instruction Delivery Format 
involves the classroom participation structures such as collaborative group work or 
individual seat work. 

Academic language exposure. Our definition of Academic Language Exposure is 
informed by recent theoretical and empirical work on language acquisition and learning (see 
e.g. Bailey, et al. 2007; Scarcella, 2003; Schleppegrell, 2004). We define ALE as opportunity 
to acquire the linguistic features through which content is typically taught, specifically the 
extent to which students are exposed to the academic vocabulary and grammatical structures 
specific to scientific contents, and the diverse range of language functions used in academic 
settings (see e.g., Bailey et ah, 2007). The survey contained three ALE sections that 
distinguished among three dimensions of ALE: ALE Instructional Strategies, ALE 
Instructional Emphases, and Academic Language Functions. ALE Instructional Strategies 
refers to the instructional strategies used to promote ALE and included scaffolding 
techniques used to support student responses during instruction, and opportunities to witness 
and participate in authentic scientific discourse (e.g., classroom visits from scientists, mock 
debates). ALE Instructional Emphases focuses on teachers’ support for the development of 
student abilities in linguistic areas related to content learning (e.g., development of 
specialized science vocabulary, conveying science facts, and reading comprehension and 
writing skills in science). Einally, Academic Language Functions covers the different ways in 
which language is used in the science classroom (i.e. description, explanation, definition and 
prediction of scientific material and ideas). 

Instrumentation 

The OTL/ALE survey. Our final instrument (the OTL/ALE survey) was informed by 
the frameworks for OTL and ALE described above, and previous work on measuring OTL 
and instructional practices at NCES and CRESST (Brewer & Stasz, 1996; NCES, 2006; 
Borko, Stecher, et. al. 2005; Boscardin et ah, 2004). Initial drafts of the OTL/ALE survey 
were reviewed by a small number of teachers and ELL experts who provided comments on 
the content, clarity, comprehensiveness, meaningfulness, and formatting of the surveys. The 
surveys were then modified based on the feedback provided by these reviewers to develop 
the final pilot survey. The final draft of the survey with nine sections and 103 items in total is 
presented in Appendix A: two sections (29 items) covered the OTL construct; three sections 
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(24 items) captured assessment practices; one section (19 items) inquired about ELL- specific 
instructional practices; finally, three sections (31 items) involved aspects of instruction 
related to ALE. Lor each item, two response columns were offered side-by-side for ELL and 
Non-ELL students respectively (with the exception of the ELL-specific strategies). In 
addition, the survey collected demographic information from the teachers, information about 
their professional background, and the classroom and school context. 

OTL/ALE observation protocol: The classroom observation instrument was refined 
from the Academic Language Exposure Checklist (ALEC, Bailey, et al. 2007) developed at 
CRESST and tried to mirror the content of the OTL/ALE survey to the extent possible (See 
Appendix B). Lor rater training, we used existing videotaped science lessons as examples; 
raters first watched the videotaped lessons and independently rated the lesson using the 
observation instrument. The independent ratings were then compared between raters and 
major disagreements discussed and resolved. 

Each of the five teachers included in observations took part in three tasks. They first 
completed the teacher OTL/ALE survey. They were then observed teaching their science 
lessons in their classroom by two trained observers on two occasions. The observers 
collected OTL/ALE data using the observation protocol. Linally, the teachers participated in 
a short debriefing interview during which they were asked to comment on the survey and 
help us improve it to better capture the experiences of ELL students in their classrooms. 
Teachers commented on clarity, burden, redundancy and overlap, and on aspects of 
OTL/ALE missing or not adequately captured in the survey. (See Appendix C). 

Reliability of the Observation Data 

Lor the classroom observations, if the independent ratings differed by more than one 
point, the two observers discussed the disagreement to reach a consensus. The consensus 
rating was entered alongside the original ratings for posterior reliability calculations. 

To investigate the reliability of observational data, we calculated two types of rater 
agreement indices: within-one-point agreement and exact agreement. We first computed both 
inter rater agreement indices separately for each item; then we calculated average indices for 
each section of the survey and an overall index across sections. Table 1 presents these 
average agreement indices by section and overall indices for the survey as a whole. Within- 
one-point agreement was high (> 0.80) on average in all sections of the observation protocol. 
Lor exact agreement, there was greater variation across sections with agreement averaging as 
high as .98 and as low as .47 for specific sections. Detailed results for within-one-point 
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agreement at the item level are presented in the tables of Appendix E. In general the results 
give confidence that consistency in teacher ratings across observers was adequate. 

Table 1 

Exact Agreement and Within-One-Point Agreement on Observation Protocols (« = 10). 



Exact agreement Within-one-point agreement 



Section 


Min 


Max 


Mean 


SD 


Min 


Max 


Mean 


SD 


Instructional activities 
& configurations 


0.88 


1.00 


0.98 


0.05 


0.88 


1.00 


0.98 


0.05 


Instructional emphases 


0.25 


0.80 


0.54 


0.19 


0.70 


1.00 


0.90 


0.10 


ELL instructional 
practices 


0.33 


0.95 


0.66 


0.17 


0.67 


1.00 


0.89 


0.12 


ALE instructional 
strategies 


0.14 


0.80 


0.47 


0.22 


0.71 


1.00 


0.89 


0.11 


ALE instructional 
emphases 


0.20 


1.00 


0.55 


0.27 


0.80 


1.00 


0.92 


0.08 


Academic language 
functions 


0.29 


0.92 


0.58 


0.20 


0.57 


1.00 


0.87 


0.12 


Overall 


0.43 


0.87 


0.63 


0.14 


0.85 


0.98 


0.92 


0.05 



Note. ELL = English language learner, ALE = Academic language exposure. 

Analytic Methods 

We address the research questions outlined above through a mixed- methods approach 
combining quantitative and qualitative methods to provide complementary perspectives from 
the different sources of data. To address the first research question we performed a series of 
paired mean comparisons to investigate differences the kinds of OTL/ALE that teachers 
report offering to EEE and non-EEE students (to adjust for inflation in error rates when 
comparing means for more than 100 items we adopted a conservative alpha level of 
significance of 0.025.) All descriptive and inferential analyses were carried out using SPSS 
v.16.0 (SPSS, 2007). 

The second research question involves conducting factor analyses to identify 
underlying constructs in the various OTE and AEE scales, and identifying areas of overlap 
among items for streamlining the survey. We used Comprehensive Exploratory Eactor 
Analysis (CEEA; Browne, Cudeck, Tateneni, & Mels, 2008) to extract principal components 
for each scale; we then reanalyzed scales deemed multidimensional using OES extraction 
(CE-Quartimax oblique rotation with Kaiser weights), choosing solutions by considering 
substantive interpretation and model fit (Chi-Squared, RMSEA, and TEI indices.) A second 
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round of factor analysis then investigated the dimensionality of the instructional practices 
captured by the OTL and ALE indicators constructed in the first step. 

To investigate our third research question (whether raters and teachers reported similar 
levels of OTL/ ALE in the classroom), we computed a series of item- level paired mean 
comparisons between survey and observation data for the five classrooms that participated in 
observations. Lor each item in the survey the values reported by teachers were compared to 
the average rating over raters and observations. Clearly, the small sample of teachers does 
not allow for meaningful significance testing and thus these data are used in a descriptive 
sense only. 

Addressing the final research question involved using case studies, debriefings, teacher 
observations, and review of open-ended comments on the survey to provide additional 
insight and context for interpreting the statistical analyses. Case studies are particularly 
useful for exploring hitherto poorly understood behaviors in real-life contexts by drawing on 
multiple sources of evidence such as observations, documents, and interviews (Yin, 2006). 
By focusing more closely on the interactions of teachers and students in the classroom, our 
case studies aim to provide a deeper understanding of the individual ways in which teachers 
provide students with opportunities to learn science and academic language, and the extent to 
which the survey might be able to capture these teaching practices accurately and validity. 
The quantitative and qualitative results finally inform a series of recommendations for future 
refinement of the survey. 

Results 

Sample Descriptives 

The 53 teachers who responded to the survey were generally well qualified, with an 
average of 8 years of experience teaching science, 4 years at their grade level. Twenty-three 
teachers had a Bachelor’s degree and 27 had a Master’s degree. About 40% of respondents 
(22) held degrees in Education, 22% (12) in the Humanities, 16% (9) in the Social Sciences, 
and the rest in other areas including the Physical Sciences and Mathematics. All but one 
teacher was fully credentialed, and 47 held an Elementary credential. Ten teachers held ESL 
credentials and six held a bilingual credential. Most of the teachers reported being proficient 
speakers of English only, but 16 were also fluent in Spanish and 3 were also fluent in other 
languages. 

There were large variations in in-service experiences reported by the teachers. The 
number of science curriculum and science methods courses ranged from 0-40, with an 
average of 5.6 curriculum courses, and 3.4 methods courses, respectively. General teaching 
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methods in-service courses ranged from 0-60 and averaged 14.3 (SD = 16.8) courses. 
Finally, the number of ELL methods in-service courses ranged from 0-50, averaging 9.6 
(SD = 11.4) courses. 

Respondents reported teaching three science lessons per week on average, each lasting 
approximately 50 minutes, with a range of 1-5 lessons lasting 20-90 minutes each. The 
average class size was 26.4 students (SD = 7.4), split relatively evenly between boys (12.6) 
and girls (14.06); on average each classroom had 20 socioeconomically disadvantaged 
students (or about 75%) and 14 English language learners (more than 50%). Notably, ELL 
students comprised only 28% and 13% of the general population of 4th grade students in 
California and Colorado respectively (California Department of Education, 2008; Colorado 
Department of Education, 2009); because our study (and our survey) targeted OTL/ALE for 
ELL students specifically, it is not surprising that teachers who volunteered to participate had 
high proportions of ELL students in their classrooms compared to the general population. 
There was also considerable variability in student performance across classrooms, with 
teachers reporting between 0-80 percent of students performing at a Proficient or Advanced 
level in language arts (M = 22%, SD = 22), between 6-78 percent of students at a Basic level 
(M=35%, 5D=18), and between 0-90 percent of students Below Basic or Ear Below Basic 
(M=41%,5D = 25). 

Research Question # 1: Teacher Self-Reported Practices with ELL and Non- ELL 
Students 

The tables in Appendix D present descriptive statistics for all sections of the OTL/ALE 
survey separately for ELL and non-ELL students. A first noteworthy pattern is that teachers 
by and large report near identical instructional practices with ELL and Non-ELL students: of 
103 OTL and ALE items, only 9 differed significantly for ELL and Non-ELL students 
(at a = 0.025.) Lor both items that inquire about the amount of time spent on instructional 
activities, and those asking about emphasis of those activities, teacher responses suggest that 
their practice does not differ systematically for ELL and non-ELL students. Nevertheless, 
there are a few interesting exceptions where significant differences were detected. With 
respect to general opportunities to learn, teachers reported that ELL students work more one- 
on-one with the teacher or as an aide than their non-ELL peers (question 17). Most 
interestingly, teacher reports reflect a different focus in the evaluation for ELL students: 
specifically, in evaluating the achievement of ELL students, teachers assigned less 
importance to understanding of scientific concepts, use of scientific vocabulary, knowledge 
of scientific facts, and progress relative to class than they did when they evaluated non-ELL 
students (Question 20). This contrasts with effort, participation, behavior, or “prior ability” 
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which were considered similarly for ELL and non-ELL students. Thus, while teachers 
reported that ELL students generally receive the same kind of OTL/ALE in the classroom, 
the foci of evaluation suggest these students may not be expected to demonstrate the same 
levels of mastery of complex knowledge as non-ELL students. This may be of particular 
interest considering teacher answers about standards and expectations for students in 
Question 23 in the survey: 49.1% of teachers reported adapting standards to accommodate 
students of differing abilities, and 37.7% reported making exceptions or special needs 
students including ELLs. Overall only 9.4% of teachers hold the same standards for all 
students without exceptions, while 90.6% makes some kind of allowance for individual 
students. 

While most teachers (over 80%) reported that ELL students do not receive science 
instruction in their primary language, they also report very little emphasis on “English-only” 
pedagogical approaches. Linally, in terms of Academic Language Exposure, teachers report 
less emphasis on developing ELL students’ scientific vocabulary and writing skills (Question 
29) compared to non-ELLs. Teachers’ reports also suggest less emphasis on reading 
comprehension, conveying, and drawing connections among scientific concepts (these 
differences are not significant at the 0.025 level however). Teacher reports did not differ 
significantly for ELL and non-ELL students in the remaining sections of the survey. 

Research Question #2: Dimensions of Opportunity to Learn and Academic Language 
Exposure 

The next step in the analysis involved use of factor analytic techniques to analyze 
patterns of inter-correlation on teacher reports of instructional practice to identify potential 
OTL and factors {dimensions) underlying the survey items and importantly, the degree 
of overlap between these OTL and ALE dimensions. Table 2 summarizes the results of 
exploratory factor analyses conducted for each section of the survey. As discussed in the 
previous section and as shown in Appendix D, teacher reports of OTL/ALE for ELL and 
non-ELL students were nearly identical for a large majority of items; the few exceptions 
involved mostly practices related to student evaluation and will be discussed below. We thus 
discuss the analyses and results for the ELL group only: Table 2 summarizes the results of 
the factor analysis carried out for each section (see Appendix E for detailed results, including 
item loadings and factor intercorrelations.) As shown in the table, we identified four factors 
underlying the Instructional Activities items in the survey (Question 17) capturing 
traditional, reform- oriented, and hands-on approaches to instruction: Moreover, the factors 
capture relatively independent facets of instruction, with factor intercorrelations ranging from 
0.11 to 0.33. Previous studies investigating instruction in science and mathematics have 
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similarly identified so-ealled reform-oriented and traditional instruetional praetiees (see e.g. 
Hamilton and Martinez, 2007). This extends to Instructional Emphasis (Question 18) where 
we identify instructional practices focusing on low and high levels of cognitive complexity 
(i.e. memorization, vs. understanding and connections; the correlation among the latter two 
dimensions is high at 0.50). Finally, we found unidimensional factors capturing teachers’ 
approaches to classroom assessment (Question 19) and teacher perceptions of the usefulness 
of standardized test scores (Question 22). Teachers who do more assessment of one kind tend 
to also do more of the others ; similarly, teachers who find test scores very useful for one 
purpose tend to find them useful for other purposes too. 

Teacher reports of ELL-specific instructional practices (Question 27) cluster around 
four factors that encompass activities directed at providing support for (a) content learning, 
(b) student strategies, (c) oral language strategies, and (d) supplemental material and 
background. Interestingly a single unidimensional factor captures the frequency of AEE- 
focused instructional strategies (Question 28). On the other hand, these activities can 
emphasize general academic language or specialized scientific language (Question 29), and 
although these two factors are strongly correlated (0.66) a unidimensional solution could be 
appropriate (indeed, 58% of variance is explained for by the 1st factor). Finally, teacher use 
of academic language functions (Question 30) clusters together functions focused on 
organizing information, providing information, and higher order thinking processes; again, 
these three factors are highly correlated with each other, suggesting that teachers who put 
more emphasis on one kind of language function tend to also do so for other functions (and 
implying a unidimensional solution might not be unjustified). 

As mentioned earlier, the results of factor analysis of OTE and AEE items for the most 
part apply equally to FEE and non-EEE students. The one exception to this pattern are 
teacher responses concerning the importance of various criteria for evaluating students 
(Question 20). As shown in Table 3, in evaluating non-EEE students, teachers consider 
indicators of content mastery on one hand (scientific vocabulary, facts, and concepts), and on 
the other a combination of behavioral indicators (i.e. effort, conduct, participation) and 
relative comparisons to other students in the class and state. However, with FEE students, 
teachers seem to consider relative progress together with absolute judgments of mastery of 
scientific contents and concepts. The findings could suggest that teachers adapt their 
expectations and judgments of mastery with consideration of FEE students’ relative status. 



Note that this assessment factor does not include standardized tests or informal (on the fly) student 
questioning. The factor comprises structured classroom assessment practices like quizzes, exams, peer 
assessment activities, etc). 
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which would be consistent with teacher reports that they make allowances for students with 
special needs in the previous section. 



Table 2 

Factor Analysis of Survey Items 




table continues 
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Table 2 {continued) 



Factors 

extracted 


Survey item 


Factors 

extracted; 

4 


Enalish Language Learner Instructional Practices (027) 

(5^2^01 = 147.73, RMSLA = 0.094, TLI = 0.91) 

Support content learning: Scaffolding, Student-Teacher Interactions, Paraphrasing, Wait 
Time, Feedback 

Student strategies: Student-Student Interactions, Peer-Assessment, Self-Assessment 
Oral Language Strategies: Clarify vocabulary. Adapt Speech, English only. Translation, 
Allow Primary Language, Use Primary Language, Practice English. 

Materials/Background: Supplementary Materials, Adapt Text, Adapt Tests, Cultural 
Background 


Factors 

extracted; 

1 


Academic Language Exposure Instmctional Strategies (028) 

(X^4 = 16.35, RMSLA = 0.057, TLI = 0.99) 

ALE Strategies: Links, Scaffolding responses. Scaffolding expectations. Clarify concepts. 
Scientific Discourse, Language Objectives, Practice Academic Language 


Factors 

extracted; 

2 


Academic Language Exposure Instructional Emphases (029) 

(X ^26 = 59.22, RMSLA = 0.157, TLI = 0.89; 57% variance in 1st factor) 

General Academic language: General Vocabulary, Grammar, Listening, Reading, Essay 
Writing 

Specialized Academic language: Science Vocabulary, Scientific Writing, Convey Concepts, 
Use Evidence, Draw Connections 


Factors 

extracted; 

3 


Academic Language Lunctions (030) 

(X ^52 = 53.47, RMSLA = 0.023, TLI = 1.00; 58% variance in 1st Lactor) 

Sorting/organizing information: Classifying, Comparing/Contrasting, Sequencing, 
Enumerating 

Providing information: Explanations, Descriptions, Definitions, Sequencing Steps, Labs, 
Higher order thinking: Causal Reasoning, Predictions, Generalizations, Inferences, Hypothesis 



Next we analyzed the patterns of relationship between the factors identified in Table 2 
to investigate whether common dimensions may underlie OTL and ALE instructional 
practices; (i.e. whether OTL and ALE are distinct features of science classrooms or can be 
seen as part of broader dimensions of instruction). We created indicators as the average of the 
items associated with each OTL and ALE factor identified in the survey (see Table 2 for the 
list of indicators and items). We then conducted factor analyses using these indicators as the 
input variables. There is moderate support for the hypothesis of a common classroom 
practice dimension underlying the OTL/ ALE indicators: 45% of the total indicator variance 
is accounted for by the first factor in a Principal Component Extraction. Perhaps not 
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surprisingly, this suggests that teachers who tend to report higher levels in the instructional 
indicators we have termed OTL also tend to report high levels of ALE. However, the patterns 
of interrelationship suggest that some aspects of instruction are more closely related than 
others. Specifically, the factor analysis results in Table 4 suggest that the indicators cluster 
around three factors that respectively reflect traditional approaches to instruction, reform 
(sometimes called constructivist) instructional practices, and ELL-specific practices. 
Interestingly, language mode practices initially identified as reform-oriented (textbooks, 
presentations and writing) here cluster together with traditional instruction. 

Table 3; 



Factor Analysis. Focus of Student Evaluation (Question 20) 





ELL students 




Non-ELL 




x"i9 = 35.1, 
RMSEA = 0.12, 
TLI = 0.92 


%"i9 = 33.3, 
RMSEA = 0.12, 
TLI = 0.94 


Locus of Evaluation 


Behavior 


Content 


El 


E2 


Achievement/progress rel. to class 


0.15 


0.41 


0.39 


0.13 


Achievement/progress rel. to state 


0.29 


0.21 


0.44 


0.09 


Achievement/progress rel. to prior ability 


0.79 


-0.11 


0.79 


-0.12 


Effort 


0.83 


0.00 


0.87 


-0.05 


Class participation 


0.83 


0.11 


0.93 


-0.02 


Behavior/conduct 


0.66 


0.16 


0.67 


0.17 


Using scientific vocabulary 


0.1 


0.70 


0.21 


0.62 


Knowing scientific facts 


-0.18 


1.06 


-0.08 


1.05 


Understanding of scientific concepts 


0.26 


0.51 


0.38 


0.42 



Note. ELL = English language learner. 



Under this scenario, an emphasis on academic language is one aspect or one dimension 
of a broader set of reform-oriented instructional practices used by science teachers in their 
classrooms; these practices emphasize understanding, connections, and higher order levels of 
cognitive complexity and use of technology. Table 4 also presents a solution where only two 
factors are extracted from the indicators; in that case ELL-specific instructional practices 
would be grouped together with traditional instruction (while the two models fit the data 
similarly well, interpretation is cleaner with three factors). While EEE-specific practices are 
by necessity remedial from the standpoint of language, this pattern could be additionally 
worrisome in suggesting that teachers who emphasize EEE-instruction most (i.e. those with 
the most EEE students in their science classrooms) may also privilege traditional 
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instructional approaches and target lower level cognitive skills, thus directly impacting the 
kinds of educational experiences of ELL and non-ELL students in those classrooms. 



Table 4 

Factor Analysis; Opportunity to Learn (OTL), English Language Learner (ELL), and Academic Language 
Exposure (ALE) Instructional Practices 





2-Eactor solution 

X"i03 = 196.5, 
RMSEA = 0.132, 
TLI = 0.86 




3-Factor solution 

X\8 = 168.2, 
RMSEA = 0.132, 
TLI = 0.86 


Instructional Practice Indicator 


Reform 


Traditional 


Reform 


Traditional 


ELL 


OTL 11 (Traditional) 


-0.25 


0.72 


-0.28 


0.51 


0.37 


OTL 12 (Technology) 


0.45 


0.33 


0.43 


0.17 


0.28 


OTL 13 (Language modes) 


0.21 


0.24 


0.13 


0.42 


-0.05 


OTL 14 (Hands-on, group) 


0.12 


0.22 


0.05 


0.36 


-0.02 


OTL21 (Connections) 


0.74 


0.13 


0.69 


0.19 


0.07 


OTL22 (Rote/memorization) 


0.06 


0.47 


0.02 


0.38 


0.22 


OTL23 (Understanding) 


0.70 


-0.09 


0.66 


0.04 


-0.05 


ELLl 1 (Support x content) 


0.22 


0.44 


0.26 


-0.05 


0.58 


ELL 12 (Student strategies) 


-0.01 


0.73 


-0.01 


0.28 


0.60 


ELL13 (Oral language) 


-0.01 


0.43 


0.05 


-0.24 


0.74 


ELL14 (Materials/background) 


0.01 


0.64 


0.02 


0.24 


0.52 


ALE 11 (ALE Strategies) 


0.54 


0.45 


0.50 


0.27 


0.34 


ALE2 1 (General language) 


0.32 


0.41 


0.16 


0.86 


-0.14 


ALE22 (Specialized language) 


0.67 


0.25 


0.59 


0.4 


0.02 


ALE3 1 (Sorting/organizing) 


0.99 


-0.24 


0.95 


-0.06 


-0.11 


ALE32 (Providing info) 


0.86 


-0.06 


0.88 


-0.13 


0.12 


ALE33 (Higher order functions) 


0.75 


0.21 


0.73 


0.14 


0.2 



OTL/ALE and ELL certification. Teachers with ELL certification in theory could 
have different conceptions about the performance of ELL students, and the best ways of 
adapting instruction to better suit the needs of this group. Table 5 presents descriptive 
statistics and mean comparisons results for the OTL and ALE factors constructed in the 
previous step as reported by teachers with and without ELL certification. As before, the small 
convenience sample (here we compare groups of 17 and 36 teachers respectively) greatly 
compromises not only power but clearly also generalizability; these results can thus only be 



16 













interpreted as preliminary and suggestive. The eomparisons yielded only two signifieant 
albeit interesting differenees (at alpha 0.05): in evaluating the achievement of ELL students, 
teachers without ELL certificates reported giving considerable more attention to behavioral 
outcomes than teachers who hold ELL certificates. In addition, ELL-certified teachers 
reported more frequent use of Oral Language techniques, including clarifying vocabulary, 
adapting speech, translation, primary language use, and practicing English, to assist ELL 
students in their classrooms. 

Table 5 



Opportunity to Learn (OTL) and Academic Language Exposure (ALE) by English Language Learner (ELL) 
Certification 



Item 


ELL instruction 


ELL certificate 
in = 17) 


No ELL certificate 
(« = 36) 


T-test 


Mean {SD) 


Mean (SD) 


(/>value) 


Q17E1 


OTL 11 (Traditional) 


2.39(1.01) 


2.48 (0.90) 


0.75 


Q17E2 


OTL 12 (Technology) 


1.66 (0.72) 


1.83 (0.51) 


0.38 


Q17E3 


OTL 13 (Language modes) 


1.98 (0.77) 


1.68 (0.84) 


0.20 


Q17F4 


OTL 14 (Hands-on, group) 


3.24 (0.86) 


3.19(0.80) 


0.87 


Q18E1 


OTL21 (Rote/memorization) 


3.18 (1.01) 


3.55 (0.68) 


0.19 


Q18F2 


OTL22 (Understanding) 


2.12(0.99) 


1.79(1.00) 


0.27 


Q18F3 


OTL23 (Connections) 


3.49 (1.23) 


3.56 (0.90) 


0.82 


Q19F1 


ASS 11 (Assessment) 


2.37 (0.90) 


2.08 (0.75) 


0.25 


Q20F1 


EVAll (Behavioral) 


3.59 (1.23) 


4.23 (0.67) 


0.05 


Q20F2 


EVA12 (Content) 


3.19(1.03) 


3.43 (0.78) 


0.40 


Q22F1 


TSTll (Usefulness of scores) 


2.51 (1.59) 


3.23 (1.14) 


0.12 


Q27F1 


ELLl 1 (Support x content) 


4.31 (0.62) 


4.17 (0.70) 


0.46 


Q27F2 


ELL 12 (Student strategies) 


3.33 (1.29) 


2.93 (1.19) 


0.29 


Q27F3 


ELL13 (Oral Language) 


2.76 (1.38) 


1.86 (1.42) 


0.04 


Q27F4 


ELL 14 (Materials/background) 


2.84(1.03) 


3.09(1.11) 


0.43 


Q28F1 


ALE 11 (ALE strategies) 


3.70 (0.79) 


3.87 (0.77) 


0.45 


Q29F1 


ALE21 (General language) 


3.88 (0.81) 


3.84 (0.77) 


0.87 


Q29F2 


ALE22 (Specialized language) 


3.48 (0.89) 


3.60 (0.73) 


0.63 


Q30F1 


ALE31 (Sorting/organizing) 


3.74 (1.17) 


4.01 (0.69) 


0.40 


Q30F2 


ALE32 (Providing info) 


3.41 (1.11) 


3.82 (0.62) 


0.17 


Q30F3 


ALE33 (High-order functions) 


3.24(1.22) 


3.33 (0.75) 


0.76 
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Overlap among items. Appendix G presents detailed results showing the correlations 
between the items in each of the OTL/ALE scales presented in Table 2. Examination of these 
correlations suggests areas where teacher self report seem to provide redundant information 
and can help us streamline the survey by reducing the current number of individual items 
(103). In analyzing these results for potential candidates for consolidation we considered 
pairs or groups of items with correlations over 0.60, and a substantial degree of conceptual 
overlap. Eor example, the item- level correlations for Question 18 point to considerable 
overlap among Relevance, Application, and Connections; these three items are closely 
connected conceptually as well and could be encompassed in a single item targeting teachers’ 
efforts to relate science to the real world and to students’ personal experiences. Similarly, 
the correlation between Background and Interest is high; teachers seem to have interpreted 
Background in direct relation to the students’ own background notions of science, which 
made it very closely related to efforts to develop their Interest in science; these items could 
either be combined into one, or further clarified what Background scientific knowledge 
means. Teacher reports in Question 18 also cluster together knowing science facts and 
understanding science concepts, apparently combining the two as higher forms of learning 
and differentiating them from memorization. Einally a high correlation suggests that teachers 
do not differentiate between inquiry and lab activities; these can either be clarified to clearly 
distinguish between specific lab skills and scientific method, or combined into a broader item 
encompassing scientific investigations. 

Peer-assessment and self-assessment activities (Question 19) also correlate strongly 
and could also be consolidated into a single item. Similar patterns of high correlation were 
observed in Question 20 (areas of emphasis for assessment) among items addressing Effort, 
Participation, and Behavior, and emphasis in Science Vocabulary and Science Facts', these 
items could be combined into one. Teachers’ responses in Question 22 suggest high overlap 
among items that capture the usefulness of test scores teacher reports indicate tests are 
similarly useful for Achievement, Instruction, Feedback, and Identifying areas of strength', 
finally, a very high correlation exists between Class work and Homework. Of the list of oral 
language strategies in Question 27 allowing students to use and using primary language 
during instruction exhibit a considerable degree of overlap; similarly high overlap was 
observed among scaffolding responses, scaffolding expectations in Question 28. Under 
Question 29 Grammar, Reading and Listening could potentially be captured through a single 
item in a streamlined version of the survey. Einally, a large degree of overlap was observed 
between a number of language functions in Question 30: classifying, comparing, and 
sequentially organizing information could be consolidated into a single item; also, a high 
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correlation exists among explanation, description, labeling and organizing in sequential 
steps. Finally, high overlap exists among items that capture Inference and Hypothesis. 

Overall, an examination of the patterns of item intercorrelation suggests several areas 
where the survey might be streamlined considerably; The net number of items could be 
reduced by as many as 30-35 items for a reduction of about l/3’^‘^ of the survey). 

Question # 3: Correspondence between OTL/ALE in Surveys and Observations 

As described in the methods section, prior to investigating agreement between surveys 
and observations, we investigated the consistency of observation ratings by examining inter- 
rater agreement indices across observers and observation occasions and at both the item and 
survey section level). Agreement within- 1 point was consistently high across sections of the 
observation protocol (see Table 1), suggesting that our observation-based ratings capture 
teacher practice with a reasonable degree of consistency (i.e. reliability). 

Appendix F presents detailed results describing the extent of correspondence between 
teacher self reports of OTL/ALE and observer ratings. The tables show the average observer 
ratings of teachers’ instructional practices across observations, and compare these to the five 
teachers’ own reports of their approach to instruction in the survey. In addition, the tables 
show the degree of agreement (within one) between teacher and observer reports (inter rater 
observation agreement is also shown for comparison). 

As a general rule, the results indicate that teachers generally reported much higher 
levels of OTL/ALE than classroom observers. Agreement was higher in terms of general 
instructional activities (Question 17); however, the items in this scale were recoded to a 0-1 
scale in the survey to match the observation protocol and thus agreement here is at a much 
coarser level (i.e. the presence or absence of a certain practice in any kind or amount in a 
given classroom). In all the remaining sections of the survey and protocol, the results reflect 
lower ratings from observers than reported by teachers. This was the case across the board in 
the sections of Instructional Emphasis (mean observer rating of 1.69, compared to an average 
of 3.40 in teacher reports), AEE strategies (mean rating 2.47, mean teacher report 3.74), AEE 
instructional emphasis (2.15 vs 3.86) and academic language functions (1.63 vs. 3.86). The 
results with EEE-specific practices follow the same pattern (1.45 vs. 3.66) with the exception 
of use of English-only instruction, for which observers ratings indicate a much more 
pervasive practice (mean: 4.2 equivalent to a rating just under always/a lot) than reflected in 
teacher reports (mean: 1.8 equivalent to a little). Teacher reports appear particularly 

3 

Teachers who rated themselves between 0 and 2 were recoded as 0 and teachers who rated themselves 
between 3 and 5 were recoded as 1 . 
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unreliable for this item: four of the five had reported that students did not receive any 
instruction in their native language (question 23) and that textbooks were available in English 
only (question 26). 

Strikingly, across sections, observers ratings were 1.88 on average, compared to 
average teacher reports of 3.70. Clearly, the results can only be considered as preliminary 
and should be interpreted with great caution, the comparisons are based on a very small 
sample of five teachers, preventing generalization of the results, and more detailed analysis 
of the measurement properties of the observation protocol through advanced psychometric 
techniques (i.e. Generalizability Analysis) to identify sources of variance in ratings of teacher 
practice. Thus, we can only speculate as to whether the disagreement reflects different 
understanding of the items by teachers and observers, limited number of classroom visits, 
teacher over- or under-reporting of instructional activities and emphases due to memory or 
social desirability effects, or a combination of all these. Researchers concerned with issues of 
reliability and validity in the measurement of instruction and classroom practice have 
identified instances of each of these factors (see e.g. Shavelson, Webb, Burstein, 1985; 
Kennedy, 1991). Nevertheless, the consistency in the degree and direction of the differences 
does seem to at least provide strong suggestive evidence of a specific kind of disagreement 
between teachers and observers, which should be considered for further exploration of 
teacher practices as they relate to OTL and ALE for ELL students. Specifically, teachers 
report that their ELL and Non-ELL students are exposed to nearly identical levels of 
OTL/ ALE in their classrooms, but their reports of amount, emphasis, and kinds of instruction 
are not confirmed by observers who visited the classrooms on two occasions, and who 
generally report lower average levels on most instructional practice items for both ELL and 
non-ELL students. 

Question # 4: Insights for improving the Survey; Qualitative Data 

The five classrooms observed are described as individual case studies that integrate the 
multiple sources of data we collected on each of them. The multiple sources included the two 
observation sessions that yielded four complete observation protocols from two observers at 
each session, the teachers’ survey responses, the teachers’ open-ended comments to the 
survey, if any, and the debriefing interviews for each teacher once the observations were 
completed. Two sets of two classrooms represent high-contrast cases in terms of the number 
of ELL students they educate, with one additional classroom containing close to an equal 
number of ELL and Non-ELL students. 
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The two classrooms with low numbers of ELLs (taught by Mrs. Troy and Mr. 
Grahame) were in an elementary school that has just 18% Latino students enrolled, with 12% 
of students school-wide participating in the free or reduced-price lunch program, and only 
7% school-wide identified as ELL students (statistics reported by the Standardized Testing 
and Reporting [STAR] program of the California Department of Education, 2008). The 
school’s Academic Performance Index (API) score of 887 for 2008 far exceeded the 
California statewide target of 800."^ On the other hand, the two classrooms with high number 
of ELLs (taught by Ms. Gomez and Ms. Llosa) were from a school that enrolls 99% Latino 
students, all are in the free and reduced-price lunch program, and 90% are identified as ELL 
students. The school’s API of 734, while not yet reaching the State target, exceeded the 2008 
growth target set for the school by 54 points (expected growth was 6 points). Linally, the 
classroom with approximately equal ELLs and Non-ELLs (taught by Ms. Sato) was located 
in a school with 89% Latino students, 62% of students school-wide enrolled in the free or 
reduced-price lunch program, and 41% of students school- wide are identified as ELL 
students. The school’s API of 724 exceeded the 2008 growth target (five points) set for the 
school by one point. 

Clearly, the three classrooms that contain the largest number of ELL students in these 
case studies are located in schools that have greater economic disadvantage and lower overall 
academic performance than the two classrooms with fewer numbers of ELL students. 
However, this is not a result of poor classroom selection but of the existence of few schools 
that have high numbers of ELL students also reporting low numbers of students participating 
in a free or reduced-priced lunch program and a high API score. Often ELL students are from 
immigrant families with lower than average incomes, and they tend to attend schools with 
lower overall academic performance (Lry, 2008). Our understanding of these case study 
classrooms will therefore need to be tempered by the social and economic disparities 
experienced by students across these different schools. (Just one each of the low- and high- 
ELL classrooms are presented below; the remaining two classrooms are presented in 
Appendix H). 

Science Instruction in Mrs. Troy’s Classroom (Low # of ELL Students). 

Background. Mrs. Troy has been teaching for 10 years which is just above the mean 
for the survey sample. She has a Master’s degree in Education and Theatre and holds an 
elementary teaching credential. She has been teaching science at the 4th grade for just 2 years 
which puts her below half the number of years of experience of the teachers in the survey 

4 

While the State target was 800, the 2008 State average API for elementary schools was in fact 776. 
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sample. Mrs. Troy has also had fewer specialized training courses (i.e., Science and ELL 
methods) than the sample overall, with just two to three courses of each. However, she 
reports attending “countless” general pedagogy and teaching methods courses. She is not 
proficient in any language other than English. 

Mrs. Troy’s classroom comprises 30 students, thus larger by four students than the 
survey average. She declined to report the numbers of boys versus girls and our observation 
protocol did not explicitly solicit this information. However, observers do not recall an 
obvious imbalance in the number of boys and girls for this classroom. This teacher also did 
not report the percentage of students who are at or above “Proficient” or fall at or (far) below 
“Basic” on the California Standards Test (CST) for English language arts (LEA). Mrs. Troy 
reported that no students came from economically disadvantaged backgrounds which is 
consistent with the STAR program demographics described above. This class has just two 
(7%) ELL students. According to the debriefing interview with Mrs. Troy, one is at a high 
level and the other is at a moderate level of English proficiency. In terms of science 
instruction, the class receives three to four lessons per week. A typical lesson varies from 
45-60 minutes. Our observation sessions were based on two 45-minute lessons with two 
different pairs of observers. Lessons took place in a purpose-built science lab at the first 
observation and in the regular classroom at the second observation. The lab classroom had a 
large lab bench at the front of the room for teacher demonstrations and six work tables with 
high stools around which students could form small groups. In her debriefing interview, Mrs. 
Troy reported that the two lessons we observed were typical of her instruction throughout the 
school year. 

Opportunity for science learning. When we observed the science lessons in the late 
spring of 2008, Mrs. Troy reported that most of the California Standards for Science had 
been covered by either herself or her co-teacher (i.e., another case study participant teacher 
Mr. Grahame) who specialized in the areas of Geology and other Earth sciences. She was 
half way to completing the standards associated with electricity after which all the science 
content standards for 4th grade would have been covered. Much of the sixth strand of the 
standards that focuses on the scientific method was reported to be covered with every science 
lesson (e.g.. Science Content Standard 6.c. “Eormulate and justify predictions based on 
cause-and-effect relationships.”). During our observations, Mrs. Troy taught the properties of 
helium, carbon and hydrogen atoms (i.e., neutrons, electrons and protons) at the first session 
and the effects of electrically charged objects at the second session. We unanimously saw 
being used or available for use a variety of resources including science books, worksheets, 
kits, magazines (e.g., the National Geographic), computer and video resources, concept 
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posters and science vocabulary charts, with the exception of a teaching assistant and district 
lesson plans. Indeed, Mrs. Troy reported having very well-resourced science classes, only 
lacking a teaching assistant and district level lesson plans. 

All four observers noted a variety of classroom activities taking place, including 
teacher directed instruction, students’ conducting hands on activities involving making model 
representations and discussing science in small groups, and culminating in completing 
worksheets individually. During these science classes, we consistently noted strong emphases 
on science conceptual understanding, basic science facts, background knowledge, conducting 
inquiries, and developing students’ interest in science. For example, we witnessed the 
explicit encouragement of hypotheses testing amongst the students as they worked in small 
groups to build models to represent the different atoms they were assigned. However, we 
consistently (across observers and observations) saw less emphasis on relevance of science to 
society, on drawing connections between concepts and on applications to everyday life. Mrs. 
Troy herself reported frequent use of a comparable wide repertoire of instructional practices 
(e.g. lecturing, use of worksheets, hands-on activities, discussing science, creating graphic 
representations and working in pairs or groups) that emphasize conceptual understanding, 
conducting inquiry, basic science facts, background knowledge, and developing students’ 
interest in science etc., with indeed a lesser emphasis on relevance of science to society, on 
drawing connection between concepts and on applications to everyday life. Our observations 
also bore out little or no emphasis by Mrs. Troy on test- taking skills and rote memorization 
of previous test questions. We witnessed no separate treatment of ELL students and Mrs. 
Troy reported that she makes no distinctions in her teaching practices with ELL and Non- 
ELL students. 

The only assessment practices observed were question and answer routines initiated by 
the teacher. In fact, Mrs. Troy’s self-reported assessment practices focus on formative 
assessment approaches using student responses to her questions, along with peer and self 
assessment, and student work in order to monitor achievement in science. In terms of what 
she finds important to evaluate in science, Mrs. Troy reported that achievement relative to the 
rest of the class, state standards and a student’s previous performance are all very important 
in evaluating students. Understanding scientific concepts was also very important in her 
opinion; other factors such as effort, class participation, behavior and knowing science facts 
were only slightly less important aspects for evaluation. 

ELL-specific science instruction: Observers consistently reported a narrow repertoire 
of possible ELL instructional strategies, with reliance on scaffolding student understanding 
of English via modeling and use of graphics and by providing many opportunities for student 
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to teacher interactions that encouraged elaborated responses. Observers noted that the teacher 
used an English-only approach to instruction, never using a student’s primary language, or 
incorporating primary language materials. In her responses to her survey, Mrs. Troy reported 
teaching the same science content to ELL and Non-ELL students. She did, however, report 
that although the same academic standards apply for everyone, she makes exceptions for 
students with special needs such as limited English proficiency. Her ELL specific practices 
include pairing her ELL students with a native English speaker, and contrary to our 
observations she reported frequently using a multitude of other ELL instructional strategies 
such as use of extra wait time, supplementary ELL materials, and paraphrasing. During her 
debriefing interview Mrs. Troy also indicated that she would check in with the moderate- 
level proficiency ELL student which was not a practice expressly stated amongst the options 
on the survey. As an English-only speaker, Mrs. Troy pointed out that she cannot use any 
student primary language strategies in her science teaching or provision of texts etc., yet she 
rated herself low on use of an English-only approach which may suggest ambiguity with this 
item or at least a different interpretation from that of the observers. 

Academic language exposure. The observers were generally in strong agreement that 
Mrs. Troy quite frequently used many of the academic English instructional strategies on the 
observation protocol, particularly scaffolding techniques such as providing model responses 
to her questions to make her expectations explicit (e.g., “My hypothesis is... and why I think 
so....”), as well as provided opportunities for students to practice academic uses of language 
orally and in writing. During both observations, we noted moderate to high emphasis on 
developing student abilities across most domains of academic English including listening and 
reading comprehension skills in science. We did not observe extended (essay) writing in 
science. There was a relatively strong emphasis across the two sessions on students being 
able to convey basic facts, as well as explain and justify scientific ideas using evidence. 
Indeed, the use of language was frequently for higher-order thinking skills such as prediction, 
inference making, causal reasoning and hypothesis generation and less often used for less 
cognitively demanding tasks such as labeling and classifying science phenomena, although 
this may have been more pronounced during the lab session. 

These observations are largely echoed in Mrs. Troy’s survey responses. She reported 
that she very often uses all the academic English strategies except authentic uses of science 
or tandem explicit language objectives which she rated as moderately used only. Like the 
observations, she reported that her teaching emphasizes student development of the higher- 
order language abilities related to communicating science (i.e., conveying, explaining and 
justifying scientific notions and ideas) and that she placed slightly less emphasis on 
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developing specific language skills such as science vocabulary and reading. She also reported 
an emphasis on science essay writing, which was however not confirmed by the observers. In 
terms of academic language functions, Mrs. Troy did not discriminate among them very 
much, rating them all as (very) frequently used. 

Summary. Overall, the observer ratings bare close resemblance to those of Mrs. Troy’s 
reported practices on the survey with the exception that the teacher survey ratings were 
almost always slightly higher than the observer ratings. Contrary to our observations, Mrs. 
Troy reported frequently using a multitude of ELL instructional strategies. However, there 
was greater agreement between our observations of academic language exposure and Mrs. 
Troy’s reported use of academic English instructional strategies and emphases, with the 
frequent use of language for higher-order thinking skills. In her debriefing interview Mrs. 
Troy indicated that she thought the survey captured her instructional practices in science very 
well. However, she reported that it wasn’t clear what she was supposed to enter for the 
survey item on assessment that requested percentages of students scoring at different levels 
on their prior CST ELA. 

Science Instruction in Ms. Gomez’s Classroom (High # ELL Students): 

Background. Ms. Gomez has been teaching for 4 years which is half the average 
number of years found in the survey sample overall. She has a Master’s degree in Education 
and holds an elementary teaching credential. She is proficient in Spanish and English. She 
has been teaching science at the 4th grade for 3 years which is just a year short of the average 
for the survey sample. Ms. Gomez did not report the number of in-services courses she might 
have taken. 

There are 25 students in Ms. Gomez’s classroom which is close to the norm for the 
survey sample. She has many more girls (16) than boys (9). All 25 students have ELL status 
and all are reported to be from economically disadvantaged backgrounds, both of which, 
recall, are consistent with the demographics for this school which is located within an 
industrial neighborhood of a large inner-city. This teacher also did not report the percentage 
of students who are “Proficient” or fall at or (far) below “Basic” on the CST for ELA. The 
class receives just two to three science lessons per week typically lasting between 45-60 
minutes each. Our observations were actually based on one 70-minute lesson and one 60- 
minute lesson with one observer consistent across the two pairs of observers. Lessons took 
place in the regular classroom which was decorated to represent an under sea world and 
contained learning centers (i.e., dedicated areas of the classroom) set up for science, reading. 
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etc. In her debriefing interview, Ms. Gomez described her usual practices and the two 
sessions were typical of this routine. 

Opportunity for science learning. The science curriculum at Ms. Gomez’s school has 
been created by teachers working in conjunction with the California Science Center and 
educational consultants. The curriculum covers different topics of science in-depth at 
different grade levels rather than revisiting the same topics each year. Therefore the 
California Standards for Science that focus on rocks and minerals had been covered during 
2nd grade, and electricity was the focus of a future grade. When we visited in the early spring 
of 2008, the class had completed all the standards that were the target of 4th grade and at the 
first session were focusing on representing living organisms through various media. At the 
second session, the focus was on ecosystems and conservation of the ocean in particular. 

With the exception of the teacher showing an illustration of coral from a science book 
and some scanned sheets of pictures from which the students could model coral, we saw no 
other use of resources. Some other kinds of resources were visibly available for use, 
including science magazines, lab materials, computer and video resources, student-made 
concept posters and science vocabulary charts. This inventory of resources is somewhat 
inconsistent with Ms. Gomez’s report on the survey, particularly the availability of a teaching 
assistant, ESL/Bilingual aide, district lesson plans and worksheets which we had understood 
to be lacking from direct questioning during the classroom visits. 

Both sessions followed a similar instructional routine with the teacher first providing 
instruction and discussing science at length with the students as a whole. The students then 
conducted hands on activities individually (session one) or in small groups (session two) to 
create play-dough and wire models of different types of coral and a poster about saving the 
oceans, respectively. The two sessions we observed had different emphases with the living 
organisms class (session one) focusing on conducting inquiry, knowing science facts, and 
building on background knowledge primarily, and the ecosystems class (session two) 
focusing on relevance of science to society and learning how scientific concepts apply to 
everyday life. Understanding key science concepts was inconsistently rated by observers. 
However, all observers unanimously agreed that both classes placed a very strong emphasis 
on developing children’s interest in science. 

Ms. Gomez reported a comparable repertoire of instructional practices on the survey. 
Moreover, at her debriefing interview she reported “a typical lesson consists of a read aloud 
or some form of text information that leads to a representation of what they have learned.” 
She placed a strong emphasis on all aspects of learning (which at least covers the different 
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emphases reported across the two sessions) and included a moderate emphasis on test-taking 
skills and rote memorization of previous test questions which we did not witness first hand. 
There was strong consensus between the two observers and between the observers and her 
survey report that Ms. Gomez focuses on formative assessment approaches using student 
responses to her questions, along with student work in order to monitor achievement in 
science. She reported that it is very important to evaluate a student’s progress relative to 
previous performance and that it is also very important to evaluate understanding of scientific 
concepts, effort, class participation, and use of scientific vocabulary. Ms. Gomez had access 
to both state and district test scores for her students and reported finding these assessment 
data largely very useful for all the purposes listed on the survey, including to assess students’ 
achievement and progress, adapt instruction and provide feedback to students 

ELL-specific science instruction: The repertoire of possible ELL instructional 
strategies observed during science instruction was largely limited to the frequent use of 
opportunities for student-to-teacher and student-to-student interactions that encouraged 
elaborated responses. Some use of scaffolding via use of visuals and diagrams was also 
consistently observed, as was paraphrasing and use of extra wait time for students to respond. 
Ms. Gomez appeared to use an English-only approach to instruction, never using a student’s 
primary language, nor incorporating primary language materials 

Even though Ms. Gomez is currently teaching ELL students only, she reported that she 
would teach the same science content in the same manner to both ELL and Non-ELL 
students. However, she reported adapting the standards to accommodate students of differing 
abilities. In contrast with our observations, Ms. Gomez reported frequently using all the listed 
ELL instructional strategies with the exception of allowing students to use their primary 
language to formulate questions or herself using students’ primary language to clarify their 
queries which she rated as moderate. The only rating that is consistent with those of the 
observers is the high rating given to the English-only approach. But this is seemingly 
inconsistent with the teacher’s own reported primary language ratings mentioned above. 
Again, this may suggest the item is being interpreted idiosyncratically. 

Academic language exposure. The two sessions yielded very different ratings by the 
observers, with the class on living organisms and building representations of coral largely 
showing little academic English instruction. Observers disagreed about the one strategy that 
might have been frequently used, that was scaffolding techniques to make expectations 
explicit such as providing model responses. Both observers noted that there was also only 
one prevalent emphasis during academic English instruction and that was developing student 
ability in specialized scientific vocabulary. In terms of academic language functions we 
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unanimously saw only evidence of a narrow repertoire mostly used to describe scientific 
objects with lesser amounts of classifying and comparing/contrasting scientific processes and 
objects. 

In contrast, during the second session observers consistently rated academic English 
instructional strategies as occurring far more frequently, particularly the teacher making 
explicit links between new concepts and student background experiences, scaffolding 
techniques to support student responses as well as making expectations clear and providing 
opportunities for students to clarify key concepts. We consistently noted a relatively strong 
emphasis on developing student abilities in listening comprehension skills in science, in 
explaining and justifying scientific ideas using evidence, and in drawing connections among 
students’ ideas. Academic language functions most frequently observed focused on higher- 
order thinking skills such as prediction, inference making, causal reasoning and hypothesis 
generation as well as on some less cognitively demanding tasks such as enumeration and 
sequencing scientific information. In contrast, Ms. Gomez reported frequently to very 
frequently employing all listed academic language instructional strategies. She did not 
discriminate among the student abilities that her teaching emphasizes, rating them all as 
strong, and she rated all academic language functions as frequently to very frequently used. 

Summary. In capturing Mrs. Gomez’s science instruction practices, the observer 
ratings and survey ratings are very similar with the notable exceptions of the reporting of 
some key resources available in the classroom, and the teacher’s reported moderate emphases 
of some low-level learning goals (e.g., test preparation) that we did not observe. In contrast 
with our observations of a narrow repertoire, Ms. Gomez reported frequently using all the 
listed ELL instructional strategies with the exception of strategies using students’ primary 
language. The two observation sessions revealed very different academic language 
exposures, but even taken together they do not cover the same types or amount of academic 
language exposure as reported by Ms. Gomez. In her debriefing interview Ms. Gomez 
suggested the survey would benefit from questions about teacher attempts to integrate 
additional subjects with science, as well as questions that prompted for the kinds of science 
projects assigned to students. 

Science Instruction in Mrs. Sato’s Classroom (Equal # ELL and Non-ELL Students). 

Background. Mrs. Sato was the most senior of the case study teachers. She has been 
teaching for 24 years, but only teaching science at the 4th grade for the past 2 years. She has 
a Master’s degree in administration and holds both elementary and ESL teaching credentials. 
However, she is not proficient in any language but English. Mrs. Sato reported taking five in- 
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service courses on science content, three for Science teaching methods and general 
pedagogy, and two for ELL teaching methods. 

There are 30 students in Mrs. Sato’s classroom, 5 students more than the survey 
average. There are 18 girls and 12 boys in the class. Six students are reported to come from 
economically disadvantaged backgrounds, and this class has 13 (43%) ELL students. The 
majority of students (80%) are at the below “Basic” or far below “Basic” levels on the CST 
for ELA. A further 14% are at “Basic” and just 6% are at or above the “Proficient” levels. 
The class is one below the norm for the survey with receiving just two science lessons per 
week. Lessons were reported to be 40 minutes in length. However, the two observations 
lasted 50 minutes. The same two researchers observed the classroom on both occasions. 

Opportunity for science learning. Mrs. Sato was in the process of teaching the 
properties of rocks and minerals at the first observation session and other Earth science 
concepts at the second session in early spring, 2008. When she completed the survey in the 
late fall of 2007, Mrs. Sato planned to cover the majority of the California Standards for 
Science having covered only Standard l.a . (designing and building simple series and parallel 
circuits). The first session included experimentation with minerals (e.g., discovering their 
properties, such as graphic producing black marks on paper), and the second session focused 
on creating a diagram of Earth’s layers. The first session used a wider range of resources, 
including science books, science kits, lab materials, concept posters and science vocabulary 
charts. The second session employed just concept posters and vocabulary charts. Computer 
and other technology types were not used at either session but were available in the 
classroom. A teaching assistant, ESL/Bilingual aide and district lesson plans were not 
available. These are extremely consistent with Mrs. Sato’s survey responses. Mrs. Sato 
reports having a reasonably well-resourced science class (has available but never uses 
computer technologies) but in her debriefing interview notes that the first session was not 
typical in that she did not have the necessary materials for minerals experimentation and had 
to provide her own. 

The two sessions were initiated with a similar instructional routine with the teacher first 
providing instruction and discussing science with the students as a whole. At the first session 
students were also referred to a page of text in their science textbooks. The students then 
conducted a hands-on activity in small groups to carry out their experiments with minerals. 
In session two, they completed a worksheet by creating and coloring a diagram of Earth’s 
layers. The observers consistently reported that the two sessions had similar emphases on 
knowing basic science facts, building on background knowledge and developing children’s 
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interest in science. The observation of the experiment also had a strong emphasis on how 
science concepts apply to everyday life. 

Mrs. Sato reported a relatively limited set of instructional practices that are consistent 
with our observations of her classes, with reading a science text book, hands-on activities and 
working in pairs or groups taking place the most frequently but nevertheless just once per 
week. However, in contrast to our observations, she only reported low to moderate emphases 
on all the targets of instruction and consistent with our observations placed no emphasis on 
using technology as a scientific tool. 

We did not observe any assessment practices during the two science classes but Mrs. 
Sato reported using most frequently weekly assessments such as checklist and rubrics, as 
well as formative assessment approaches using student responses to her questions, along with 
self assessment, and student work in order to monitor achievement in science. She reported 
as primarily only moderately important all the possible areas of evaluation in science class 
(e.g., progress relative to the class, effort, using science vocabulary, etc.). However, she 
reported finding standardized test scores that she has access to as useful to a large degree for 
all the purposes included on the survey, but mainly useful for assessing students’ 
achievement and progress, adapting instruction, provide feedback to students and parents and 
for identifying areas of strength and weakness. 

ELL-specific science instruction. Both observations consistently revealed a narrow 
repertoire of possible ELL instructional strategies, with reliance on scaffolding student 
understanding of English via modeling and use of manipulatives, by providing many 
opportunities for student to teacher interactions that encouraged elaborated responses, by 
paraphrasing and use of extra wait time. Observers noted that the teacher used an English- 
only approach to instruction, never using a student’s primary language, or incorporating 
primary language materials. Although Mrs. Sato has relatively large numbers of both ELL 
and Non-ELL students in her class, she reported the same responses for teaching ELL 
students as for teaching Non-ELL students. Nor does she group her students differently 
because of language proficiency, rather she groups her students by how well they get along 
together (i.e., for behavior reasons). She reported that while the same academic standards 
apply for everyone, she makes exceptions for students with special needs such as limited 
English proficiency. In her debriefing interview, Mrs. Sato reported that both her ELL 
students and her Non-ELL students have low proficiency in English and she has to go slowly, 
use a lot of repetition and lots of defining of even simple words. 
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There are both consistencies and inconsistencies in what observers noted and Mrs. 
Sato’s reported use of ELL instructional strategies. While there is a high degree of 
correspondences for use of scaffolding and providing opportunities for student-to-teacher and 
student- to-student interactions that encourage elaborated responses, as well as deliberate use 
of wait time and use of paraphrasing at the second session, ratings for other strategies showed 
less correspondence. The most noteworthy are Mrs. Sato’s report of using some primary 
language during instruction and correspondingly virtually no English-only approach. 
Observers saw no primary language usage and consistently rated as “always” on the item 
about the use of an English-only approach. 

Academic language exposure. At both sessions, the observers were in agreement that 
Mrs. Sato moderately used many of the academic English instructional strategies, particularly 
making explicit links between new concepts and students’ background and past learning, 
using scaffolding techniques to support student responses as well as making teacher 
expectations explicit. She provided frequent opportunities for students to practice academic 
uses of language both orally and in writing. During session one and the mineral 
experimentation, Mrs. Sato also frequently made language objectives explicit in addition to 
science knowledge objectives. During both observations, we consistently noted a strong 
emphasis on developing student specialized vocabulary in science, as well as moderate 
emphases on listening comprehension skills and on students being able to convey basic facts. 
Language Mrs. Sato used, most frequently functioned to describe, compare/contrast, label, 
and define, and in the case of diagramming Earth to also explain geologic processes, rather 
than function in support of higher-order thinking skills, such as prediction or hypothesis 
generation. This is surprising given the experiment conducted on different types of minerals. 

Although Mrs. Sato’s reported academic English instructional strategies largely mirror 
those of the observers in terms of type and frequency, there are inconsistencies between the 
observations and Mrs. Sato’s report on the degree of emphases of instructional practices and 
which academic language functions are employed. She did not discriminate across emphases 
at all, reporting all as moderate. She also reported moderate usage of language functions and 
discriminates among them only slightly by reporting a little less use of classifying, 
comparing/contrasting, and causal reasoning. In both her response to the open-ended prompt 
for further comment on the survey and in her debriefing interview, Mrs. Sato reported that 
she has less time to teach science than she used to have. Some of the items mentioned on the 
survey she felt she simply did not have time for. This is despite the recent addition of science 
to the required annual assessments (NCLB, 2002). She also reported that the ELD program 
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that she is required to teach incorporates science but the content is not linked to the 4th grade 
California Science Standards. 



Summary. Overall, the limited set of instructional practices that were observed at both 
sessions are consistent with Mrs. Sato’s survey report. However the narrow repertoire of ELL 
instructional practices observed is not replicated by the survey responses in which Mrs. Sato 
reports frequently using a variety of strategies including student primary language strategies 
while she herself reported not being proficient in any language but English nor has the help 
of a bilingual aide. Measures of academic language exposure varied in the degree of 
agreement between observers and survey responses, with strong agreement between observed 
and reported frequency and type of academic language strategies, but far less agreement on 
the emphases of instructional practices and academic language functions due to a lack of 
discrimination between items on the survey 

Summary and Implications for Survey Development 

In this section we summarize the results of our study and discuss the lessons learned 
and implications for further development and validation work. We presented the results of 
piloting work aimed at refining and validating a survey-based measure of Opportunity to 
Learn (OTL) and Academic Language Exposure (ALE) in Science. Our study was based on a 
small pilot sample for surveys (53 teachers) and a smaller subsample for classroom 
observations (5 teachers) and thus the results can only be interpreted as preliminary and 
suggestive of next validation steps for the survey instrument. However, some of the results 
point to areas that seem to hold potential for contributing to our understanding of ELL 
instruction, and of the educational experiences of ELL students in general, and thus warrant 
future investigation. 



Summary of Quantitative Results 

1. Teachers largely report very similar instructional practices with ELL and Non-ELL 
Students. However, ELL students may not be expected to demonstrate the same 
levels of mastery of complex knowledge as others: our sample of teachers reported 
giving less attention to understanding concepts, using scientific vocabulary, 
knowing scientific facts, and learning progression relative to other students in the 
class when evaluating ELL students. 
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2. Our comparison of survey reports for teachers with and without ELL certification 
yielded few significant differences (though they were affected by low power due to 
small sample size). However, the differences observed are interesting: ELL- 
certified teachers report less emphasis on behavioral outcomes when evaluating 
ELL students, and greater emphasis on helping ELL students through oral language 
interventions (e.g. clarifying vocabulary, adapting speech translation, or allowing 
primary language). 

3. Teacher reports of their own instructional practices differ consistently and 
substantially from those of observers who visited their classrooms. Specifically, 
teachers consistently report more frequent use of a large number of instructional 
approaches, and greater emphasis on many aspects or dimensions of instruction 
than was apparent to the observers during their classroom visits. Granted, 
differences in frequency could partly reflect the periods covered by the reports (all 
year for teacher reports in the survey, two visits for the observers); however, these 
differences are consistent, if not greater, for items that capture instructional 
emphases on OTL and ALE (as opposed to frequency). Overall, the results seem to 
suggest a social desirability effect that results in teacher reports of their own 
instructional practice that are consistently more positive than reported by observers. 

4. Item Lactor Analysis of the within sections of the survey yielded a series of 
OTL/ALE factors reflecting instructional practices and emphasis. Some of these 
mirror the results of previous studies that have reported factors capturing traditional 
and reform-oriented instruction. The LA results were similar for ELL and Non-ELL 
students with one exception: in evaluating ELL students’ teachers consider the 
student behavior on one hand, and relative performance and understanding of 
scientific on the other. With Non-ELL students, mastery of substantive content is 
considered on its own, and relative comparisons and behavioral outcomes are a 
separate dimension. It is not clear how this difference should be interpreted, 
however, since teachers may combine absolute and relative criteria for different 
purposes in evaluating ELL students (i.e. a comparison to other ELLs could 
compensate for low absolute performance; but a comparison to the rest of the class 
may tend to penalize ELL students). 

5. Lactor Analysis of the OTL/ALE indicators suggests that instructional practices 
targeting Academic Language (ALE) tend to occur and be emphasized in 
classrooms alongside reform-oriented instructional practices such as a focus on 
understanding and making connections among concepts or using technology to 
support learning. This suggests that the nature of ALE practices is closely related to 
instruction that focuses on and targets higher-level cognitive processes and 
understanding of contents. The close relationship between ALE and some features 
of quality instruction in science suggests that both ELL and Non-ELL students 
might benefit when teachers make ALE part of his or her instructional practices in 
the classroom. 
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6. Factor Analysis also indicates that ELL-specific instruction is a separate dimension 
of instruction; however, instructional practices targeting ELL students are more 
closely related to so called traditional instructional approaches, and instruction that 
targets low level learning (e.g. memorization of facts) than they are to reform- 
oriented approaches. This trend would seem to be exemplified by a comment from a 
teacher in the open ended section of the survey who said about ELL instruction: 
“Unfortunately I don't have low ELL students, so I don't have to do much extra for 
them. I do have a low class in general, so I use some ELL techniques in my overall 
teaching.” While perhaps not surprising these kinds of perceptions (if widespread) 
would nonetheless be worrisome as they inherently equate instructional approaches 
that address language limitations with instruction that targets low achieving 
students. 

7. The patterns of intercorrelation suggest that the survey may be strea ml ined 
considerably by consolidating as many as one third of items that provide redundant 
information. 

Lessons Learned from the Case Studies and the Survey Responses 

While inconsistency between observations and teacher survey responses could be a 
function of the small number of science lessons observed (i.e. any narrow repertoires 
witnessed could reflect a lack of opportunity to see a teacher’s full range of strategies in just 
two sessions), the quantitative results overall suggest that social desirability may play a role 
in teacher reports of their own practice. Another reason for inconsistency may be a lack of 
discrimination in rating items (i.e., circling all items as high emphasis without realizing the 
“directionality” of some items is reversed); or even misinterpretation of items. 

(a) Lor example, “English-only” was meant as indicative of complete reliance on 
English without support for students in their primary language. Observers saw little use of 
the students’ primary language and rated all teachers highly on this item. However, teachers 
seemed to have misunderstood this item. Two teachers rated themselves as low on this item 
which obviously didn’t correspond with the observer ratings. One of these teachers reported 
using some primary language during instruction although she also reported being only 
proficient in English and not having a bilingual aide. Another rated herself high on English- 
only which is consistent with the observer ratings but inconsistent with her own report of 
using primary language in instruction. Linally, the quantitative analysis also indicate some 
confusion or inconsistency in teacher reports for this and related items: most teachers 
reported not offering ELL students any science instruction in their primary language, but they 
also reported very little emphasis on an “English only” pedagogical approach. 
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(b) Interestingly, in the three ease studies, there was greater eorrespondenee between 
observer and teaeher reports of instruction for science and academic language than for ELL 
strategies. We are examining how these sections were constructed to determine whether they 
may have primed teachers to over-report on their usage of ELL instructional strategies. Items 
for ELL practices appear to consist of more concrete practices (e.g., “Use of supplementary 
ELL materials including dictionaries and thesauruses”), whereas the science and academic 
language practices appear to be more abstract and require more careful interpretation, such as 
the determining the importance to students for “Learning about the relevance of science to 
society” and emphasizing “Explicit links between new concepts and students background 
experiences and past learning. ” 

(c) The case studies also revealed that teachers tended not to discriminate on the 
frequency of the different academic language functions, while the observers reliably detected 
teaching practices that favored some academic language functions over others. In the future 
we might take one of two possible approaches: either requiring teachers rank order the 
academic language functions for importance, or giving teachers a mechanism to more 
concretely report how often they include these functions in their science classes. Lor 
example, teachers could be asked to report how often they use these functions in their lesson 
plans as part of their descriptions of science objectives or tasks. 

(d) Teacher reactions to and feedback about the survey on the open ended prompt for 
comments (Question 31) were highly informative, often helping illuminate aspects of their 
instruction or their classroom or school context that may not have come across in the survey. 
These often had direct implications for refinement of the survey. 

Many teachers felt that the survey covered their instructional practices adequately and 
there were no suggestions of important aspects of ELL instruction omitted: 

Y our survey appears to focus pretty accurately on the needs and circumstances of second 
language learners... Overall, I thought the survey was well done and covered many 
aspects of the subjects. It made me really think and reflect about my teaching of 
science... The survey was easy to complete and reminded me of many effective teaching 
practices which I can employ to promote success in my students. Thanks 

However, some comments point to potential sources of misunderstanding that could 
have resulted in inconsistencies in teacher reports: “I'm not sure the questions were direct 
enough to get the information you needed.” And “Some questions weren’t clear in regards 
to. . . I wasn’t sure what was really being asked.” 
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Other comments speak directly to the importance of the local classroom and school 
context for understanding ELL instruction in science. These ranged from the specific mix of 
ELL students to the local policy or curricular context. Many teachers suggested adding space 
for explaining and providing context for their answers. 

At our school, our science time is also our low-emergent reader’s intervention 
time.... Our school has 22 languages. (This might be a good question for Section I, 
because it makes a big difference if all if your ELLs are of the same lang.)...This survey 
could be more effective if there are more questions relating to the school where we work. 

Lurthermore, a number of teacher comments emphasize the problems in treating ELL 
students as a homogeneous group. There is great variance within ELL students in terms of 
background and performance and this can have direct implications for instruction. Moreover, 
the proportion or specific characteristics of ELL students may help explain the lack of 
variance in teacher practices observed. 

On the ELL-Specifics practices it was hard to answer because I have different levels of 
ELLs... Perhaps a question about the differing ELL proficiencies would be helpful. Also 
grouping all my ELL students together to answer several of your questions was hard. All 
students are taught very similarly - then additional instruction is given as needed. 

Some comments directly suggest a number of sensible improvements to the survey. 

There is a large gap between once a month and once a week. It makes it hard to 
choose. . .Perhaps a change or adaptation of the survey could be instead of having an ELL 
vs. Non-ELL section, maybe you could have a space for teachers to add adaptations or 
accommodations, if any, do you make for your ELL. 

The second comment in particular is thoroughly supported by our quantitative findings 
showing little distinction in teacher reports for ELL/Non-ELL students. 

Linally, a common thread was the length of the survey: (e.g., “Many of the questions 
were repetitive.” “Too Long.” “Very redundant.” “$30 is not enough money to fill out this 
long and tedious survey.”) These comments are actually quite reasonable and not surprising; 
in developing the survey we decided to include items that we considered potentially 
interesting even if we had questions about how respondents would interpret them, or if they 
appeared to overlap with other items. In our directions to the survey we forewarned the 
respondents that redundancy in items may exist in some cases, but that we were trying out 
different approaches to get at the same kinds of information. Luture versions of the survey 
can focus on those items that yielded sufficient and reliable information in our efforts to 
better understand the academic opportunities of ELL students. 



36 




To conclude, the development of valid measures of OTL/ALE could be of great help in 
studying and clarifying the relationship between key dimensions of instruction and student 
academic achievement, particularly in the case of ELL students. It is also potentially 
important as a tool for investigating instructional practice and policies that seek to improve 
the educational experiences of ELL students in the U.S. These tools can also be extended to 
understand instruction more generally as well as the experiences of English proficient/native 
speakers, who may nevertheless face inequities in Opportunity to Learn and/or Academic 
Language Exposure in their schools leading to disadvantages for academic success. 
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Appendix A: 

Teacher Survey of Science Instruction (CA Version) 

1, TEACER AND CLASSROOM INFORMATION 

1. Including this year, how many years have you been a teacher? years 

2. Including this year, how many years have you been teaching science at this grade? years 

3. What is the highest degree you have attained? (Check one) 

□ Associate □ Master □ PhD 

□ Bachelor □ EdD Other 

4. What was your major area of study for that degree? (Check one) 

□ Humanities □ Mathematics D Biological Sciences 

□ Social Sciences D Physical Sciences Other 

5. What kind(s) of teaching credential(s) do you hold? (Check all that apply) 

D Elementary D Bilingual D Emergency 

D Secondary D ESL Other 

6. How many in-service courses have you attended focusing on; 

Science contents/curriculum 

Science teaching methods 

General pedagogy/teaching methods 

Teaching methods for ELL students 

7. Are you proficient in a language other than English? (Check all that apply) 

□ No □ Cantonese □ Vietnamese 



D Spanish □ Mandarin Other 

8. Based on the CA Standards Test (CST) ELA test, what percentages of students in your science class are; 



% Below Basic or Far Below Basic 


% Basic 


% At or Above Proficient 


= 100% 









9. How many total students do you teach in this science classroom? 

10. How many girls and boys are there in your science classroom? girls boys 

11. How many of those students are English Language Learners (ELL students)? 

12. How many of your students are from economically disadvantaged backgrounds? 

13. How many science lessons do your students receive per week? 

14. How long is each science lesson? minutes 
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II. CURRICULAR CONTENT COVERAGE 



15. Please indicate below the coverage of 4th grade science standards in your class. 
(Check one for each response) 



4th Grade Science Standard 


Didn ’t 
cover and 
don’t 
plan to 
cover 


Covered 


Plan to 
cover 


Approximate 
number of 
lessons spent/to 
be spent on the 
topic 


1. Electricity and magnetism are related effects that have many useful applications in everyday life. As a basis 
for understanding this concept students will know; 


a How to design and build simple series and 
parallel circuits by using components such as 
wires, batteries, and bulbs. 


□ 


□ 


□ 


# of lessons 


b How to build a simple compass and use it to 
detect magnetic effects, including Earth's 
magnetic field. 


□ 


□ 


□ 


# of lessons 


c How electric currents produce magnetic fields 
and how to build a simple electromagnet. 


□ 


□ 


□ 


# of lessons 


d The role of electromagnets in the construction of 
electric motors, electric generators, and simple 
devices, such as doorbells and earphones. 


□ 


□ 


□ 


# of lessons 


e Electrically charged objects attract or repel each 
other. 


□ 


□ 


□ 


# of lessons 


f Magnets have two poles (north and south) and 
that like poles repel each other while unlike poles 
attract each other. 


□ 


□ 


□ 


# of lessons 


g Electrical energy can be converted to heat, light, 
and motion. 


□ 


□ 


□ 


# of lessons 


2. All organisms need energy and matter to live and grow. As a basis for understanding this concept students 
will know; 


^ Plants are the primary source of matter and 
energy entering most food chains. 


□ 


□ 


□ 


# of lessons 


b 

Producers and consumers (herbivores, carnivores, 
omnivores, and decomposers) are related in food 
chains and food webs and may compete with 
each other for resources in an ecosystem. 


□ 


□ 


□ 


# of lessons 


^ Decomposers, including many fungi, insects, and 
microorganisms, recycle matter from dead plants 
and animals. 


□ 


□ 


□ 


# of lessons 
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4th Grade Science Standard 


Didn ’t 
cover and 
don’t 
plan to 
cover 


Covered 


Plan to 
cover 


Approximate 
number of 
lessons spent/to 
be spent on the 
topic 


3. Living organisms depend on one another and on their environment for survival. As a basis for 
understanding this concept students will know; 


^ Ecosystems can he characterized hy their living 
and nonliving components. 


□ 


□ 


□ 


# of lessons 


^ In any particular environment, some kinds of 
plants and animals survive well, some survive 
less well, and some cannot survive at all. 


□ 


□ 


□ 


# of lessons 


^ Many plants depend on animals for pollination 
and seed dispersal, and animals depend on plants 
for food and shelter. 


□ 


□ 


□ 


# of lessons 


d Most microorganisms do not cause disease and 
that many are beneficial. 


□ 


□ 


□ 


# of lessons 


4. The properties of rocks and minerals reflect the processes that formed them. As a basis for understanding 
this concept students will know; 


^ How to differentiate among igneous, 

sedimentary, and metamorphic rocks by referring 
to their properties and methods of formation (the 
rock cycle). 


□ 


□ 


□ 


# of lessons 


^ How to identify common rock-forming minerals 
(including quartz, calcite, feldspar, mica, and 
hornblende) and ore minerals by using a table of 
diagnostic properties. 


□ 


□ 


□ 


# of lessons 


5. Waves, wind, water, and ice shape and reshape Earth's land surface. As a basis for understanding this 
concept students will know; 


a Some changes in the earth are due to slow 

processes, such as erosion, and some changes are 
due to rapid processes, such as landslides, 
volcanic eruptions, and earthquakes. 


□ 


□ 


□ 


# of lessons 


b Natural processes, including freezing and 
thawing and the growth of roots, cause rocks to 
break down into smaller pieces. 


□ 


□ 


□ 


# of lessons 


c Moving water erodes landforms, reshaping the 
land by taking it away from some places and 
depositing it as pebbles, sand, silt, and mud in 
other places (weathering, transport, and 
deposition). 


□ 


□ 


□ 


# of lessons 
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4th Grade Science Standard 


Didn ’t 
cover and 
don ’t plan 
to cover 


Covered 


Plan to 
cover 


Approximate 
number of lessons 
spent/to be spent 
on the topic 


6. Scientific progress is made by asking meaningful questions and conducting careful investigations. As a 
basis for understanding this concept and addressing the content in the other three strands, students should 
develop their own questions and perform investigations. Students will; 


^ Differentiate observation from inference 

(interpretation) and know scientists’ explanations 
come partly from what they observe and partly 
from how they interpret their observations. 


□ 


□ 


□ 


# of lessons 


^ Measure and estimate the weight, length, or 
volume of objects. 


□ 


□ 


□ 


# of lessons 


^ Formulate and justify predictions based on cause- 
and-effect relationships. 


□ 


□ 


□ 


# of lessons 


Conduct multiple trials to test a prediction and 
d draw conclusions about the relationships between 
predictions and results. 


□ 


□ 


□ 


# of lessons 


e Construct and interpret graphs from measurements. 


□ 


□ 


□ 


# of lessons 


^ Follow a set of written instructions for a scientific 
investigation. 


□ 


□ 


□ 


# of lessons 



III. INSTRUCTIONAL RESOURCES 

16. Are the following resources/materials/support used in your classroom? (Check one for each response) 





Yes 


No (But Available in Class) 


No (Not Available in Class) 


Science books 


□ 


□ 


□ 


Science newspapers/magazines 


□ 


□ 


□ 


Science kits 


□ 


□ 


□ 


Lab equipment/materials 


□ 


□ 


□ 


Computers 


□ 


□ 


□ 


Science-specific software 


□ 


□ 


□ 


Internet 


□ 


□ 


□ 


Video resources 


□ 


□ 


□ 


Science vocabulary charts 


□ 


□ 


□ 


Concept posters 


□ 


□ 


□ 


Procedure posters 


□ 


□ 


□ 


Dictionaries for students 


□ 


□ 


□ 


Thesaurus for students 


□ 


□ 


□ 


Teaching assistant 


□ 


□ 


□ 


ESL/bilingual aide 


□ 


□ 


□ 


District provided lesson plans 


□ 


□ 


□ 


Worksheets 


□ 


□ 


□ 


Other Ispecifv; 1 


□ 


□ 


□ 
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IV. GENERAL INSTRUCTIONAL PRACTICES 



17. Approximately how frequently do the students in your science class do the following? 
(Circle one for each response) 





ELL Students 


Non-ELL Students 


Never 


7-2 

times a 
year 


Once 

a 

month 


Once 

a 

week 


A few 
times a 
week 


Every 

day 


Never 


1-2 

times a 
year 


Once a 
month 


Once 

a 

week 


A few 
times a 
week 


Every 

day 


Listen to a lecture/ 
T eacher-directed 
instruction 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Read a science 
textbook 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Read science articles 
in magazines or 
newspapers 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Complete worksheets 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Conduct hands-on 
activities or 
manipulate materials 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Discuss science 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Homework 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Watch videos 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Present oral science 
reports 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Complete a written 
science report 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Use science-related 
software 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Use science-related 
internet resources 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Create graphic 
representations 
(drawings, diagrams, 
models, etc.) 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Work on individual 
projects 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Work on projects in 
pairs or groups 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Work in groups 
arranged by ability 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Work one-on-one 
with a teacher or an 
aide 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Other 

( ) 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 
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18. Considering the time you are able to devote to teaching science in your classroom, to what extent do your 
instmctional practices emphasize the following for students; (Circle one for each response) 



ELL Students 



No 

Emphasis 



Non-ELL Students 



Strong No 

Emphasis Emphasis 



Strong 

Emphasis 



Learning about the 



relevance of science to 
society 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Knowing basic science 
facts and terminology 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Understanding key 
science concepts 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Developing test-taking 
skills 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Memorization of previous 
test questions 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Learning how scientific 
concepts apply to 
everyday life 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Using overarching 
concepts to draw 
connections between 
different science topics 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Building on background 
scientific knowledge 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Developing laboratory 
skills and techniques 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Conducting inquiries or 
investigations in science 
(set up hypothesis, collect 
and analyze data) 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 










Developing children’s 
interest in science 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Using technology as a 
scientific tool 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Other 

( ) 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 
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19. Considering the time you are able to devote to teaching science in your classroom, approximately how 
frequently do you use or do the following to assess student achievement in science? (Circle one for each 
response) 





ELL Students 


Non-ELL Students 






1-2 


Once 


Once 


A few 






1-2 


Once 


Once 


A few 








times 


a 


a 


times 


Every 




times 


a 


a 


times 


Every 




Never 


a year 


month 


week 


a week 


day 


Never 


a year 


month 


week 


a week 


day 


Teacher-made 

quizzes 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Chapter (unit) end 
tests 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Other assessments 
(e.g., checklists, 
mhrics) 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Trial standardized 
tests or released test 
items 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Use of student 


























responses to 
questions, student 
explanations, and 
the observation of 
student interactions 
for assessment 
purposes during 
instruction 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Student work (e.g., 
science journals, 
portfolios, 
homework) 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Peer assessment 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Student self- 
assessment 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Other 

( ) 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 
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20. How important are the following for evaluating students in your science class? (Circle one for each response) 





ELL Students 


Non-ELL Students 




Not 

Important 






Very 

Important 


Not 

Important 






Very 

Important 


Achievement/ 

progress relative to the rest 

of the class 


0 


1 


2 


3 


4 


5 


0 


2 


3 


4 


5 


Achievement/ 

progress relative to State 

Standards 


0 


1 


2 


3 


4 


5 


0 


2 


3 


4 


5 


Achievement/ 
progress relative to the 
child’s previous learning (or 
performance) 


0 


1 


2 


3 


4 


5 


0 


2 


3 


4 


5 


Effort 


0 


1 


2 


3 


4 


5 


0 


2 


3 


4 


5 


Class participation 


0 


1 


2 


3 


4 


5 


0 


2 


3 


4 


5 


Behavior/conduct 


0 


1 


2 


3 


4 


5 


0 


2 


3 


4 


5 


Using scientific vocabulary 


0 


1 


2 


3 


4 


5 


0 


2 


3 


4 


5 


Knowing scientific facts 


0 


1 


2 


3 


4 


5 


0 


2 


3 


4 


5 


Understanding of scientific 
concepts 


0 


1 


2 


3 


4 


5 


0 


2 


3 


4 


5 


Other ( I 


0 


1 


2 


3 


4 


5 


0 


2 


3 


4 


5 



21. Do you have access to individual standardized test scores for the students in your class? (Check all that apply) 

□ Yes, from Statewide assessments 

□ Yes, from District assessments (e.g. benchmarks) 

□ Yes, from other assessments ( ) 

□ No (SKIP TO QUESTION 23) 
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22. How useful do you find the student standardized test scores for the following? (Circle one for each response) 





ELL Students 


Non-ELL Students 




Not 

Useful 










Very 

Useful 


Not 

Useful 








Very 

Useful 


Assess student achievement 
and progress 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Adapt/guide instruction 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Provide feedback to students 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Identify areas of strength and 
weakness 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Provide information for 
parents 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Assign additional or different 
work in class 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Assign additional or different 
homework 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Other ( 1 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 



23. What are your policies for teaching and evaluating students in your classroom? (Check one) 

□ The same academic standards apply for everyone in my classroom, no exceptions 

□ The same academic standards apply for everyone, hut I make exceptions for students with special 
needs (e.g. disahilities, limited English proficiency) 

□ I adapt the standards to accommodate students of differing abilities 

V. ELL-SPECIFIC PRACTICES 

24. Are students in your science classroom grouped differently because of language? (Check all that apply) 

□ Yes, each ELL student is paired with a native English speaker 

□ Yes, ELL students are grouped together by language proficiency level 

□ Yes, all ELL students are placed in one group regardless of proficiency and native language 

□ Yes, ELL students with the same language (e.g. all Spanish-speakers) are placed in one group 

□ No, Students are not grouped differently because of language 

25. Are ELL students given science instmction in their primary language? 

□ YES 

□ NO 

□ OTHER (Explain; ) 

26. Are ELL students given science text books in; 

□ English □ Their Primary Language 
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27. How often do you use (or to what extent do you rely on) the following English language learner (ELL) 
instructional strategies in your science class? (Circle one for each response) 





Never / 
Not at 
All 










Always/ 
A lot 


Use scaffolding techniques to support students’ 
understanding of English (e.g. think-aloud 
/modeling, graphics, realia, manipulatives) 


0 


1 


2 


3 


4 


5 


Provide opportunities for student-to-teacher 
interactions that encourage elaborated responses 


0 


1 


2 


3 


4 


5 


Provide opportunities for student-to-student 
interactions that encourage elaborated responses 


0 


1 


2 


3 


4 


5 


Provide activities for students to practice 
English using new skills, concepts, and 
vocabulary 


0 


1 


2 


3 


4 


5 


Provide opportunities for students to clarify 
English vocabulary in their primary language 


0 


1 


2 


3 


4 


5 


Provide opportunities for peer assessment of 
English proficiency 


0 


1 


2 


3 


4 


5 


Provide opportunities for student self-assessment 
of English proficiency 


0 


1 


2 


3 


4 


5 


Use supplementary ELL materials, including 
dictionaries and thesauruses 


0 


1 


2 


3 


4 


5 


Adapt my English speech and language to all 
levels of English proficiency (differentiated 
instruction) 


0 


1 


2 


3 


4 


5 


Adapt texts and assignments to all levels of 
English proficiency 


0 


1 


2 


3 


4 


5 


Use tests for ELL students with low levels of 
language demand (differentiated assessment) 


0 


1 


2 


3 


4 


5 


Identify and provide feedback about students’ 
weaknesses in English language (e.g. vocabulary, 
grammar) 


0 


1 


2 


3 


4 


5 


Use ‘"English Only' approach during instruction 


0 


1 


2 


3 


4 


5 


Use simultaneous translation of English to 
primary language during instruction 


0 


1 


2 


3 


4 


5 


Explicitly incorporate students’ home culture and 
background experiences in school activities 


0 


1 


2 


3 


4 


5 


Allow students to use their primary language in 
formulating questions 


0 


1 


2 


3 


4 


5 


Use students’ primary language to clarify 
students’ queries 


0 


1 


2 


3 


4 


5 


Use paraphrasing to make input comprehensible 


0 


1 


2 


3 


4 


5 


Use deliberate pauses (wait time) to allow 
students to respond 


0 


1 


2 


3 


4 


5 
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VI. ACADEMIC LANGUAGE PRACTICES 



Academic English language is the language (i.e., vocabulary, grammar and discourse patterns) used 
for teaching content-area knowledge. It can be specific to a single discipline (e.g., specialized 
vocabulary such as electro-magnetic, or ecosystem), or it may be more generalized across several 
disciplines (e.g., general academic vocabulary such as produce, analyze, outcome). 

28. Considering the time you are able to devote to teaching science in your classroom, how often do you 
use (or alternatively, to what extent do you rely on) the following academic English instructional strategies? 
(Circle one for each response) 



ELL Students 



Never/ 
Not at 
All 



Always 
/A lot 



Non-ELL Students 



Never / 
Not at 
All 



Always 
/A lot 



Explicit links between new concepts 
and students’ background 
experiences and past learning 

Scaffolding techniques to support 
students’ responses (e.g., provide a 
mix of challenging open-end 
questions and less challenging yes/no 
questions). 

Scaffolding techniques to make 
explicit teacher expectations for 
students’ responses (e.g., provide 
model responses, explicit teaching of 
necessary grammatical structures for 
elaborated responses). 

Providing opportunities for students 
to clarify key concepts in science 

Providing opportunities for students 
to witness the authentic discourse of 
scientists (e.g., video, classroom 
visits from botanists etc.) and 
participate (e.g., mock debates) in 
scientific discourse 

Explicit language objectives, in 
tandem with science knowledge/skill 
objectives 

Providing opportunities for students 
to practice uses of academic 
language in science (e.g., 
new general academic vocabulary 
and new scientific vocabulary, 
grammatical structures such as 
comparatives, nominalizations, and 
discourse level skills) both orally and 
in writing 



2 3 4 5 



2 3 4 5 



2 3 4 5 



2 3 4 5 



2 3 4 5 



2 3 4 5 



0 1 



2 3 4 5 



0 1 2 3 4 5 



0 1 2 3 4 5 



0 1 2 3 4 5 



0 1 2 3 4 5 



0 1 2 3 4 5 



0 1 2 3 4 5 
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29. Considering the time you are able to devote to teaching science in your classroom, to what extent do your 
instructional practices emphasize the following for students; (Circle one for each response) 



ELL Students 



No 

Emphasis 



Non-ELL Students 



Strong No 

Emphasis Emphasis 



Strong 

Emphasis 



Student ability to develop 
general academic vocabulary 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Student ability to develop 
specialized scientific 
vocabulary 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Student ability to develop 
necessary grammatical 
structures to both talk and write 
about science 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Student ability to develop 
active listening 
skills/comprehension 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Student ability to develop 
reading comprehension skills in 
science 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Student ability to develop 
scientific writing skills 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Student ability to develop essay 
writing skills using science 
topics 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Student ability to convey basic 
scientific concepts accurately 
and effectively 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Student ability to use evidence 
to explain and justify scientific 
notions and ideas 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


Student ability to draw 
connections among students’ 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 



ideas 
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30. Considering the time you are able to devote to teaching science in your classroom, indicate how often you 
used the following academic language functions (i.e., ways in which you used language for teaching 
science); (Circle one for each response) 



ELL Students Non-ELL Students 





Never / 
Not at 
All 










Always 
/A lot 


Never / 
Not at 
All 








Always 
/ A lot 


Explanation of science 
processes 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Description of objects or 
phenomena 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Definition of scientific 
vocabulary and phrases 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Classifying scientific 
phenomenon 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Comparing/Contrasting 
scientific processes or 
objects 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Causal reasoning with 
scientific phenomena 
(cause and effect) 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Sequencing or organizing 
extended scientific 
information 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Sequencing steps of 
scientific procedures or 
experiments 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Labeling of scientific 
processes or objects 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Enumeration (listing) of 
science facts and 
processes 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Prediction of scientific 
outcomes 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Generalization of 
scientific processes or 
facts to other scientific 
phenomena 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Inference making based 
on scientific knowledge 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Hypothesis-generating 
based on scientific facts 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 


Other 


0 


1 


2 


3 


4 


5 


0 1 


2 


3 


4 


5 
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OVER PLEASE 



31. As a final step we would like to ask for any thoughts, comments or suggestions you might have 
about the survey, its contents, format, or organization. One of the purposes of this project is to 
develop a survey that reflects teacher practices and student experiences in the classroom as 
accurately as possible. If you think of any ways in which the survey or any of its sections could 
be improved to better reflect classroom practice (e.g. by clarifying confusing or ambiguous 
questions, or adding aspects of teacher practice), we would be grateful if you share them with us. 



Thank you for your participation! 



Appendix B: 

Classroom Observation Protocol 

Date: 

Observer: 

School & RM# 

Grade: 



• Content Coverage 

1. Which science topics and standards were taught during the class? (See Appended list of possible 
science topics from state standards.) Please familiarize yourself with Appendix, ask teacher to clarify 
during debriefing if necessary, and record the need. 



Time: 

Teacher: 

Subject area: 

Lesson duration: 



• Classroom Physical Setting/Participation 

2.1 Using the space provided below, please draw the physical layout of the classroom setting. 



2.2 How was the classroom configured? (Check all that were observed. Add time specificity as 
record, e.g., 1/2 of session) 

□ Whole Class □ Small Groups □ Pairs 

□ Individual Work □ One-on-one with Teacher/Teacher’s Aid 



2.3 Were ELL students configured differently? If so, how did they differ (e.g. one ELL student 
paired with one English-only student, ELL students with the same first language background placed 
in one group)? Clarify with teacher any groupings that are not clear. 



2.4 What is the general participation pattern of students responding to questions in the science class? 

(Circle only one answer) 

□ Small core group participates □ Half the class participates □ Call on/encourage everyone to participate 
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• Classroom Context (If possible, please fill in the information before class starts.) 

3.1 How many total students were present in the classroom? 

3.2 How many of those students present are ELL? (Confirm with teacher) 



• Availability and Use of Instructional Resources 

4.1 Lor the following questions, please check the materials/resources that were USED in the 
classroom. (Check one) 







No 


No 


No 






(But available 


(But available 


(Not available 






in the 


based on 


based on 




Yes 


classroom ) 


teacher report) 


teacher report) 


Science books 


□ 


□ 


□ 


□ 


Science newspapers/magazines 


□ 


□ 


□ 


□ 


Science kits 


□ 


□ 


□ 


□ 


Lab equipment/materials 


□ 


□ 


□ 


□ 


Computers 


□ 


□ 


□ 


□ 


Science-specific software 


□ 


□ 


□ 


□ 


Internet 


□ 


□ 


□ 


□ 


Video resources 


□ 


□ 


□ 


□ 


Science vocabulary charts 


□ 


□ 


□ 


□ 


Concept posters 


□ 


□ 


□ 


□ 


Procedure posters 


□ 


□ 


□ 


□ 


Dictionaries for students 


□ 


□ 


□ 


□ 


Thesaurus for students 


□ 


□ 


□ 


□ 


Teaching assistant 


□ 


□ 


□ 


□ 


ESL/bilingual aide 


□ 


□ 


□ 


□ 


District provided lesson plans 


□ 


□ 


□ 


□ 


Worksheets 


□ 


□ 


□ 


□ 


Other (specify; 

) 


□ 


□ 


□ 


□ 
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• Classroom Activities & Configuration 

5.1 What activities did students engage in? (Check all that apply) 



□ 


Listen to a lecture/ Teacher-directed instruction 


□ 


Read a science textbook 


□ 


Read science articles in magazines or newspapers 


□ 


Complete worksheets 


□ 


Conduct hands-on activities or manipulate materials 


□ 


Discuss science 


□ 


Homework (Any activities except for assigning homework) 


□ 


Watch videos 


□ 


Present oral science reports 


□ 


Complete written science reports 


□ 


Use science-related software 


□ 


Use science-related internet resources 


□ 


Create graphic representations (drawings, diagrams, models, etc.) 


□ 


Work on individual projects 


□ 


Work on projects in pairs or groups 


□ 


Work in groups arranged by ability 


□ 


Work one-on-one with a teacher or an aide 



Other activities not listed above. (Please specify the new category /categories): 



Please write down any relevant information for others reading your observation, or your reflectional 
thoughts on the protocol. 



5.2 Did ELL students engage in different activities from non-ELL students? If so, what activities 
differed? 
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• Instructional Emphases 

6.1 To what extent did classroom instructional practices emphasize the following (See Appendix for 
definitions of and examples for each item): (Circle one) 







ELL Students 


Non-ELL Students 




NOTES 


No Emphasis Strong Emphasis 


No Emphasi 


s Strong Emphasis 


1 . Learning about the relevance 
of science to society 




0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


2. Knowing basic science facts 
and terminology 




0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


3. Understanding key science 
concepts 




0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


4. Developing test-taking skills 




0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


5. Memorization of previous test 
questions 




0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


6. Learning how scientific 
concepts apply to everyday life 




0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


7. Using overarching concepts to 
draw connections between 
different science topics 




0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


8. Building on background 
scientific knowledge 




0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


9. Developing laboratory skills 
and techniques 




0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


10. Conducting inquiries or 
investigations in science (set up 
hypothesis, collect and analyze 
data) 




0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


11. Developing children’s 
interest in science 




0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


12. Using technology as a 
scientific tool 




0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 



Other emphases not listed above (Please specify the new category/categories): 



Please write down any relevant information for others reading your observation, or your reflectional 
thoughts on the protocol. 
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• Assessment Practices 

7. 1 What assessment practices were used? (Check all that apply) 



□ 


Teacher-made quizzes 


□ 


Chapter (unit) end tests 


□ 


Other assessments (e.g., checklists, rubrics) 


□ 


Trial standardized tests or released test items 


□ 


Use of student responses to questions, student explanations, and the observation of 
student interactions for assessment purposes during instruction 


□ 


Student work (e.g., science journals, portfolios, and homework) 


□ 


Peer assessment (e.g. students read each other’s homework and provide feedback) 


□ 


Student-self assessment 



Other assessment practices not listed above (Please specify the new category/categories): 



Please write down any relevant information for others reading your observation, or your reflectional 
thoughts on the protocol. 



7.2 Were assessments of ELL students different? If so, how did they differ? 
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• ELL Instructional Practices 

8.1 Reflecting on the class as a whole, how often, if at all, were the following ELL instructional 
practices used? (Circle one for each item) 





Never / 
Not at All 








Always 
/ A lot 


1. Use scaffolding techniques to support students’ understanding of 
English (e.g. think-aloud /modeling, graphics, realia, manipulatives) 


0 


1 


2 


3 


4 


5 


2. Provide opportunities for student-to-teacher interactions that 
encourage elaborated responses 


0 


1 


2 


3 


4 


5 


3. Provide opportunities for student-to-student interactions that 
encourage elaborated responses 


0 


1 


2 


3 


4 


5 


4. Provide activities for students to practice English using new 
skills, concepts, and vocabulary 


0 


1 


2 


3 


4 


5 


5. Provide opportunities for students to clarify English vocabulary in 
their primary language 


0 


1 


2 


3 


4 


5 


6. Provide opportunities for peer assessment of English proficiency 


0 


1 


2 


3 


4 


5 


7. Provide opportunities for student self-assessment of English 
proficiency 


0 


1 


2 


3 


4 


5 


8. Use supplementary ELL materials, including dictionaries and 
thesauruses 


0 


1 


2 


3 


4 


5 


9. Adapt English speech and language to all levels of English 
proficiency (differentiated instruction) 


0 


1 


2 


3 


4 


5 


10. Adapt texts and assignments to all levels of English proficiency 


0 


1 


2 


3 


4 


5 


11. Identify and provide feedback about students’ weaknesses in 
English language (e.g. vocabulary, grammar) 


0 


1 


2 


3 


4 


5 


12. Use "'English Only" approach during instruction 


0 


1 


2 


3 


4 


5 


13. Use simultaneous translation of English to primary language 
during instruction 


0 


1 


2 


3 


4 


5 


14. Explicitly incorporate students’ home culture and background 
experiences in school activities 


0 


1 


2 


3 


4 


5 


15. Allow students to use their primary language in formulating 
questions 


0 


1 


2 


3 


4 


5 


16. Use students’ primary language to clarify students’ queries 


0 


1 


2 


3 


4 


5 


17. Use paraphrasing to make input comprehensible 


0 


1 


2 


3 


4 


5 


18. Use deliberate pauses (wait time) to allow students to respond 


0 


1 


2 


3 


4 


5 



Other ELL instructional practices not listed above (please specify): 



Please write down any relevant information for others reading your observation, or your reflectional 
thoughts on the protocol. 
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• ALE practices 

9.1. Reflecting on the class as a whole, indicate how often the teacher used the following academic 
English instructional strategies: (Circle one) 



ELL Students 



Never / 
Not at 
All 



Always / 
A lot 



Non-ELL Students 



Never/ 
Not at 
All 



Always / 
A lot 



1 . Explicit links between new concepts and 
students’ background experiences and past 
learning 

2. Scaffolding techniques to support 
students’ responses (e.g., provide a mix of 
challenging open-end questions and less 
challenging yes/no questions). 

3. Scaffolding techniques to make explicit 
teacher expectations for students’ 
responses (e.g., provide model responses, 
explicit teaching of necessary grammatical 
structures for elaborated responses). 

4. Providing opportunities for students to 
clarify key concepts in science 

5. Providing opportunities for students to 
witness the authentic discourse of scientists 
(e.g., video, classroom visits from botanists 
etc.) and participate (e.g., mock debates) in 
scientific discourse 

6. Explicit language objectives, in tandem 
with knowledge/skill objectives 

7. Providing opportunities for students to 
practice uses of academic language in 
science (e.g., new general academic 
vocabulary and new scientific vocabulary, 
grammatical structures such as 
comparatives, nominalizations, and 
discourse level skills) both orally and in 
writing 



0 1 



0 1 



2 3 4 5 



0 1 2 3 4 5 



0 1 2 3 4 5 



2 3 4 5 



0 1 2 3 4 5 



0 1 2 3 4 5 



2 3 4 5 



0 1 



2 3 4 5 



0 1 2 3 4 5 



0 1 2 3 4 5 



0 1 2 3 4 5 



0 1 2 3 4 5 



0 1 2 3 4 5 



0 1 2 3 4 5 



Other strategies not listed above (please specify): 



Please write down any relevant information for others reading your observation, or your reflectional 
thoughts on the protocol. 
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9.2. Reflecting on the class as a whole, indicate to what extent the teacher’s academic language 
instructional practices emphasized the following (See Appendix for examples for each item): 
(Circle one) 





ELL Students 


Non-ELL Students 




No 

Emphasis 






Strong 

Emphasis 


No 

Emphasis 






Strong 

Emphasis 


1. Student ability to develop general academic 
vocabulary 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


2. Student ability to develop specialized 
scientific vocabulary 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


3. Student ability to develop necessary 
grammatical structures to both talk and write 
about science 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


4. Student ability to develop active listening 
skills/comprehension. 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


5. Student ability to develop reading 
comprehension skills in science 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


6. Student ability to develop scientific writing 
skills 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


7. Student ability to develop essay writing skills 
using science topics 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


8. Student ability to convey basic scientific 
concepts accurately and effectively 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


9. Student ability to use evidence to explain and 
justify scientific notions and ideas 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


10. Student ability to draw connections among 
students’ ideas 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 



Other emphases not listed above (please specify): 



Please write down any relevant information for others reading your observation, or your reflectional 
thoughts on the protocol. 
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9.3 Reflecting on the class as a whole, indicate how often the teacher used the following academic 
language functions (i.e. ways in which the teacher used language for teaching science) to ELL and Non- 
ELL students: (Circle one) 





ELL Students 


Non-ELL Students 




Never/ 
Not at All 






Always / 
A lot 


Never/ 
Not at All 






Always / 
A lot 


1. Explanation of science processes 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


2. Description of objects or phenomena 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


3. Definition of scientific vocabulary and 
phrases 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


4. Classifying scientific phenomenon 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


5. Comparing/Contrasting scientific 
processes or objects 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


6. Causal reasoning with scientific 
phenomena (cause and effect) 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


7. Sequencing or organizing extended 
scientific information 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


8. Sequencing steps of scientific procedures 
or experiments 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


9. Labeling of scientific processes or 
objects 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


10. Enumeration (listing) of science facts 
and processes 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


1 1 . Prediction of scientific outcomes 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


12. Generalization of scientific processes or 
facts to other scientific phenomena 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


13. Inference making based on scientific 
knowledge 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


14. Hypothesis-generating based on 
scientific facts 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 


15. Other 


0 


1 


2 


3 


4 


5 


0 


1 


2 


3 


4 


5 
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Appended to Protocol: 

POSSIBLE SCIENCE TOPICS 

1. Electricity and magnetism are related effects that have many useful applications in everyday life. As a basis 
for understanding this concept students will know: 

a. How to design and build simple series and parallel circuits by using components such as wires, batteries, and 
bulbs. 

b. How to build a simple compass and use it to detect magnetic effects, including Earth's magnetic field. 

c. How electric currents produce magnetic fields and how to build a simple electromagnet. 

d. The role of electromagnets in the construction of electric motors, electric generators, and simple devices, 
such as doorbells and earphones. 

e. Electrically charged objects attract or repel each other. 

f Magnets have two poles (north and south) and that like poles repel each other while unlike poles attract each 
other. 

g. Electrical energy can be converted to heat, light, and motion. 

2. All organisms need energy and matter to live and grow. As a basis for understanding this concept students 
will know: 

a. Plants are the primary source of matter and energy entering most food chains. 

b. Producers and consumers (herbivores, carnivores, omnivores, and decomposers) are related in food chains 
and food webs and may compete with each other for resources in an ecosystem. 

c. Decomposers, including many fungi, insects, and microorganisms, recycle matter from dead plants and animals. 

3. Living organisms depend on one another and on their environment for survival. As a basis for 
understanding this concept students will know: 

a. Ecosystems can be characterized by their living and nonliving components. 

b. In any particular environment, some kinds of plants and animals survive well, some survive less well, and 
some cannot survive at all. 

c. Many plants depend on animals for pollination and seed dispersal, and animals depend on plants for food 
and shelter. 

d. Most microorganisms do not cause disease and that many are beneficial. 

4. The properties of rocks and minerals reflect the processes that formed them. As a basis for understanding 
this concept students will know: 

a. How to differentiate among igneous, sedimentary, and metamorphic rocks by referring to their properties 
and methods of formation (the rock cycle). 

b. How to identify common rock-forming minerals (including quartz, calcite, feldspar, mica, and hornblende) 
and ore minerals by using a table of diagnostic properties. 

5. Waves, wind, water, and ice shape and reshape Earth 's land surface. As a basis for understanding this 
concept students will know: 

a. Some changes in the earth are due to slow processes, such as erosion, and some changes are due to rapid 
processes, such as landslides, volcanic emptions, and earthquakes. 

b. Natural processes, including freezing and thawing and the growth of roots, cause rocks to break down into 
smaller pieces. 

c. Moving water erodes landfotms, reshaping the land by taking it away from some places and depositing it as 
pebbles, sand, silt, and mud in other places (weathering, transport, and deposition). 

6. Scientific progress is made by asking meaningful questions and conducting careful investigations. As a basis 
for understanding this concept and addressing the content in the other three strands, students should develop 
their own questions and perform investigations. Students will: 

a. Differentiate observation from inference (interpretation) and know scientists’ explanations come partly from 
what they observe and partly from how they interpret their observations. 

b. Measure and estimate the weight, length, or volume of objects. 

c. Eormulate and justify predictions based on cause-and-effect relationships. 

d. Conduct multiple trials to test a prediction and draw conclusions about the relationships between predictions 
and results. 

e. Construct and interpret graphs from measurements. 

f. Eollow a set of written instructions for a scientific investigation. 
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DEFINITIONS AND EXAMPLES FOR INSTRUCTIONAL EMPHASIS 



Item 


Definition 


Example 


Learning about the 
relevance of science 
to society 


Emphasizes ways in which society is 
influenced by scientific discovery and 
advances. 


How electricity affected communication; 
how medical inventions affect lifespan. 


Knowing basic science 
facts and terminology 


Emphasizes definition, memorization, 
recall, and comprehension. 


Recite and define specialized science 
vocabulary such as “photosynthesis”, 
“water cycle”, etc. 


Understanding key 
science concepts 


Emphasizes synthesis, analysis, 
evaluation, and application. 


Explain how electricity works; identify 
faulty argument; use representations to 
model scientific concepts 


Developing test-taking 
skills 


Emphasizes formal testing rules, 
conventions, and/or strategies. 


Activities that are explicitly marked as test 
prep, i.e. mimicking the testing conditions 
such as timing the students or asking the 
students to close their books, should 
receive a high rating; activities that meet 
the criteria but are not explicitly marked 
may receive a lower rating. 


Memorization of 
previous test questions 


No further definition needed. 


NA 


Learning how 
scientific concepts 
apply to everyday life 


Emphasizes connections between in- 
class activities and specific real-world 
activities. 


Explain how the electrical circuits work for 
the light-bulbs to light up when the switch 
is turned on. 


Drawing connections 
between different 
scientific concepts 


Emphasizes abstract concepts and 
processes that can be identified across 
contexts; emphasizes explanation of 
relationships between different science 
models or representations, and 
identification of patterns after data 
analysis. 


Explain the connection between 
photosynthesis and food chain. 


Building on 
background scientific 
knowledge 


Emphasizes building on or referring 
back to prior classroom knowledge, and 
using previously-learned factual or 
procedural knowledge, e.g. same 
methodology, to learn a new scientific 
concept. 


Referring back to previous lesson on cloud 
formation to explain the water cycle 
concept; applying the same steps and 
methodology, such as experimental design 
and data collection procedure, to 
conducting another science project of a 
different topic 


Developing laboratory 
skills and techniques 


Emphasizes correct laboratory 
practices, manipulation of materials, 
and technical/mechanical skills. 


How to use lab equipment such as a 
microscope 


Conducting inquiries 
or investigations in 
science (set up 
hypothesis, collect and 
analyze data) 


Emphasizes generating questions and 
devising strategies to investigate them. 


Work on a science project; plant seeds in 
different conditions, make hypothesis 
about the end product, collect growth data, 
conduct data analysis, and draw 
conclusions. 


Developing children’s 
interest in science 


Emphasizes child-appropriate activities 
and contexts that promote children’s 
engagement. 


Ties activities to pop culture, uses contests 
and games, etc. 


Using technology as a 
scientific tool 


Emphasizes the use of technology, such 
as lab equipment (but excluding simple 
lab materials such as tubes and food 
coloring) and computers, as a tool for 
doing science. (Note; As opposed to 
using technology, such as the internet, 
simply to access information.) 


Use microscope to investigate the soil 
collected from different environments; use 
GPS devices to identify or visit certain rare 
plant/animal sightings and observe long- 
term changes. 
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EXAMPLES FOR ACADEMIC LANGUAGE INSTRUCTIONAL PRACTICES 



Item 


Example 


Student ability to develop 
general academic vocabulary 


Practices that focus on teaching vocabulary that cuts across 
academic disciplines (e.g., produce, analyze, report) and 
having students produce this in their speech and writing 
and comprehend them in others’ speech and in printed 
texts 


Student ability to develop 
specialized scientific vocabulary 


Practices that focus on teaching vocabulary that is science 
discipline-specific (e.g., photosynthesis, electromagnet, 
sedimentary rock) and having students produce this in their 
speech and writing and comprehend them in others’ speech 
and in printed texts 


Student ability to develop 
necessary grammatical 
structures to both talk and write 
about science 


Practices that focus on sentence forms and constructions 
(e.g., if... then clauses to convey clausal connections; 
superlative and comparative forms such as smallest, 
smaller to convey contrasts) and having students produce 
these forms in their speech and writing and comprehend 
them in others’ speech and in printed texts 


Student ability to develop 
reading comprehension skills in 
science 


Practices that deliberately teach making meaning of 
science texts and having students use reading 
comprehension strategies as they read science texts (e.g., 
metacognitive strategies such as identifying main ideas, 
monitoring own comprehension during reading etc.) 


Student ability to develop 
scientific writing skills 


Practices that teach science writing (e.g., expository, 
informational genre), having students produce their own 
writing in a science context (e.g., learn the correct 
organization of reports, experiments, arguments etc.). 


Student ability to convey basic 
scientific concepts accurately 
and effectively 


Practices that teach the need for clear communication of 
basic science information and having students accurately 
retell basic science facts and terminology 


Student ability to use evidence 
to explain and justify scientific 
notions and ideas 


Practices that explicitly teach the composition of good 
explanations of science concepts based on evidence and 
justifications for scientific arguments and having students 
communicate their explanations and justifications 
following acceptable scientific conventions 


Student ability to draw 
connections among students’ 
ideas 


Practices that focus on helping students to make links 
between their own and each others’ scientific ideas and 
reasoning 
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Appendix C: 

Debriefing Interview 

lA. How representative of your typical instruction during the year for this science classroom 
were the two lessons we observed? (in terms of contents, instructional practices, activities, etc.) 



IB. If not, how were these lessons not representative (and why)? 



2. Is there something about the context of this classroom and its (ELL) students that we 
should know that would help us better understand your instructional practices? 



3. How well do you think the content of the survey captures your instructional practices in 
the science classroom? 



4. Are there additional things the survey should inquire about in order to get a better sense of 
teacher instruction and specifically the experiences of (ELL) students in the classroom? 
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Appendix D: 



OTL/ALE Survey Item Descriptives: Comparisons of ELL and Non-ELL students 
SECTION IV: GENERAL INSTRUCTIONAL PRACTICES (OTL) 

Instructional Activities & Configurations (Question 17) 



Approximately how frequently do the students 
in your science class do the following? 


ELL students 




Non-ELL students 


f-test 

(p-val) 


Median 


Mean 


SD 


Median 


Mean 


SD 


Discuss science 


4 


3.89 


0.8 


4 


3.92 


0.8 


NA 


Listen to lecture/teacher instruction 


4 


3.85 


1.1 


4 


3.87 


1.1 


.322 


Conduct hands-on activities/manipulate 


4 


3.64 


0.7 


4 


3.63 


0.7 


NA 


Work on projects in pairs or groups 


4 


3.23 


1.3 


4 


3.23 


1.3 


1.000 


Read a science textbook 


3 


2.98 


1.3 


3 


2.98 


1.3 


NA 


Complete worksheets 


3 


2.91 


1.0 


3 


2.92 


1.0 


NA 


Create graphic representations 


3 


2.77 


0.9 


3 


2.81 


1.0 


.420 


Homework 


2 


2.40 


1.7 


2 


2.35 


1.7 


NA 


Work in groups arranged hy ability 


3 


2.21 


1.9 


3 


2.25 


1.9 


.322 


Work one-on-one with a teacher or an aide 


2 


1.92 


1.8 


1 


1.58 


1.6 


.013 


Read science articles in magazines/newspapers 


2 


1.79 


1.3 


2 


1.77 


1.3 


.322 


Watch videos 


2 


1.62 


0.8 


2 


1.60 


0.8 


.322 


Work on individual projects 


1 


1.48 


1.2 


1 


1.48 


1.2 


1.000 


Complete a written science report 


1 


1.28 


0.9 


1 


1.28 


0.9 


NA 


Use science-related internet resources 


1 


1.19 


1.0 


1 


1.26 


1.1 


.322 


Present oral science reports 


1 


1.06 


0.8 


1 


1.08 


0.8 


.322 


Use science-related software 


0 


0.40 


0.7 


0 


0.47 


0.9 


.322 
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Instructional Emphases (Question 18) 



To what extent do your instructional practices 
emphasize the following for students 


ELL students 




Non-ELL students 


f-test 

(p-val) 


Median 


Mean 


SD 


Median 


Mean 


SD 


Developing children’s interest in science 


4 


4.32 


0.7 


4 


4.32 


0.7 


NA 


Understanding key science concepts 


4 


3.91 


0.9 


4 


3.91 


0.9 


1.000 


Building on background scientific knowledge 


4 


3.87 


1.1 


4 


3.81 


1.1 


.182 


Knowing basic science facts/terminology 


4 


3.75 


1.2 


4 


3.75 


1.1 


1.000 


Conducting inquiries or investigations 


4 


3.60 


1.3 


4 


3.58 


1.3 


.322 


Learning how science applies to everyday life 


4 


3.36 


1.2 


4 


3.36 


1.2 


1.000 


Using concepts to draw connections 


3 


3.28 


1.1 


3 


3.28 


1.1 


1.000 


Learning about the relevance of science 


3 


3.21 


1.3 


3 


3.23 


1.3 


.322 


Developing test-taking skills 


3 


2.89 


1.4 


3 


2.92 


1.4 


.322 


Developing laboratory skills or techniques 


3 


2.89 


1.5 


3 


2.89 


1.6 


1.000 


Using technology as a scientific tool 


3 


2.53 


1.3 


3 


2.51 


1.3 


.322 


Memorization of previous test questions 


1 


0.91 


1.0 


1 


0.96 


1.1 


.322 


Student Assessment (Question 19) 


Approximately how frequently do you 
use or do the following to assess student 
achievement? 


ELL students 




Non-ELL students 


f-test 

(p-val) 


Median 


Mean 


SD 


Median 


Mean 


SD 


Student work 


4 


3.72 


1.0 


4 


3.72 


1.0 


NA 


Use of student responses/explanations 


4 


3.49 


0.9 


4 


3.47 


0.9 


.322 


Other assessments 


3 


2.65 


1.1 


3 


2.65 


1.1 


NA 


Student self-assessment 


2 


1.94 


1.4 


2 


1.94 


1.4 


NA 


Teacher made quizzes 


2 


1.66 


1.0 


2 


1.68 


1.1 


.322 


Chapter (unit) end tests 


2 


1.53 


0.9 


2 


1.51 


0.9 


.659 


Peer assessment 


1 


1.51 


1.5 


1 


1.53 


1.5 


.322 


Trial standardized tests or released test items 


1 


1.06 


1.0 


1 


1.06 


1.0 


NA 
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Focus of Student Evaluation (Question 20) 



How important are the following for 
evaluating students in your science class? 


ELL students 




Non-ELL students 


f-test 

(p-val) 


Median 


Mean 


SD 


Median 


Mean 


SD 


Effort 


5 


4.38 


1.0 


5 


4.38 


1.0 


NA 


Class participation 


4 


4.15 


1.0 


4 


4.19 


0.9 


.159 


Understanding of scientific concepts 


4 


3.92 


1.0 


4 


4.06 


1.0 


.018 


Achievement/progress relative to prior ability 


4 


3.79 


1.2 


4 


3.81 


1.2 


.322 


Behavior/conduct 


4 


3.77 


1.2 


4 


3.77 


1.2 


NA 


Using scientific vocabulary 


3 


3.47 


1.0 


4 


3.70 


1.0 


.009 


Knowing scientific facts 


3 


3.34 


1.0 


4 


3.45 


1.0 


.033* 


Achievement/progress relative to state 


3 


3.29 


1.3 


3 


3.40 


1.3 


.057 


Achievement/progress relative to class 


3 


2.74 


1.6 


3 


2.92 


1.5 


.017 


Uses of Test Scores (Question 22) 


How useful do you find the student 
standardized test scores for the following? 


ELL students 




Non-ELL students 


f-test 

(p-val) 


Median 


Mean 


SD 


Median 


Mean 


SD 


Identify areas of strengths and weaknesses 


4 


3.29 


1.6 


4 


3.33 


1.6 


.160 


Adapt/guide instruction 


3 


3.26 


1.3 


3.5 


3.36 


1.2 


.044* 


Provide information for parents 


4 


3.17 


1.5 


4 


3.24 


1.5 


.083* 


Assess student achievement and progress 


3 


2.93 


1.6 


3 


3.05 


1.6 


.096* 


Provide feedback to students 


3 


2.81 


1.7 


3 


2.86 


1.7 


.160 


Assign additional or different work in class 


3 


2.62 


1.6 


3 


2.64 


1.6 


.660 


Assign additional or different homework 


3 


2.50 


1.7 


3 


2.52 


1.7 


.570 
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SECTION V: ELL SPECIFIC PRACTICES (OTL) 
ELL Instructional Practices (Question 27) 



How often do you use the following 
English language learner instructional strategies 
in your science class? 


Median 


Mean 


SD 


Use wait time to allow students to respond 


5 


4.55 


0.6 


Scaffolding techniques to support English 


5 


4.40 


0.9 


Opportunities for student/student interactions 


5 


4.36 


0.7 


Opportunities to practice English 


4 


4.11 


1.0 


Adapt speech to all levels of proficiency 


5 


4.11 


1.7 


Use paraphrasing 


4 


4.08 


0.9 


Opportunities for student/teacher interactions 


4 


4.06 


0.8 


Incorporate students culture and background 


4 


3.34 


1.5 


Adapt texts to all levels of proficiency 


4 


3.23 


1.6 


Use “English Only” 


4 


3.21 


2.0 


Supplementary ELL materials 


3 


3.08 


1.8 


Opportunities to clarify vocabulary 


3 


3.02 


1.7 


Provide feedback about students English 


3 


2.96 


1.5 


Use tests with low levels of language demand 


3 


2.70 


1.8 


Allow primary language in questions 


2 


2.36 


1.8 


Opportunities for self assessment of English 


2 


2.32 


1.6 


Opportunities for peer assessment of English 


2 


2.15 


1.7 


Use primary language to clarify queries 


2 


2.04 


1.8 


Use simultaneous translation 


0 


1.12 


1.6 
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SECTION VI: ACADEMIC LANGUAGE PRACTICES 



ALE Instructional Strategies (Question 28) 





ELL students 

1 factor 




Non-ELL students 

1 factor 


t-test 

(p-val) 


English instructional strategies? Median Mean 


SD 


Median 


Mean 


SD 


Scaffolding techniques for responses 5 


4.47 


0.7 


5 


4.47 


0.7 


NA 


Links between new concepts and background 5 


4.40 


0.7 


5 


4.40 


0.7 


NA 


Opportunities for students to clarify concepts 4 


4.33 


0.8 


4 


4.31 


0.8 


.322 


Scaffolding techniques for expectations 4 


3.98 


1.1 


4 


3.92 


1.1 


.083 


Opportunities to practice academic language 4 


3.85 


0.9 


4 


3.85 


0.9 


NA 


Explicit language objectives 3 


3.23 


1.4 


3 


3.21 


1.4 


.766 


Opportunities to witness authentic discourse 3 


2.49 


1.6 


3 


2.49 


1.6 


NA 


ALE Instructional Emphases (Question 29) 




ELL students 




Non-ELL students 




To what extent do your instructional practices 












f-test 


emphasize the following for students? Median Mean 


SD 


Median 


Mean 


SD 


(p-val) 


Ability to develop active listening skills 


4 4.26 


0.7 


4 


4.30 


0.7 


.159 


Ability to develop general academic vocab 


4 4.09 


0.7 


4 


4.11 


0.8 


.569 


Ability to draw connections among others ideas 


4 3.85 


0.8 


4 


3.92 


0.8 


.044 


Ability to use evidence to justify science 


4 3.83 


1.0 


4 


3.83 


1.1 


1.000 


Ability to develop grammatical structures 


4 3.79 


1.0 


4 


3.91 


1.0 


.109 


Ability to develop reading comprehension skills 


4 3.74 


1.1 


4 


3.79 


1.1 


.083 


Ability to convey basic scientific concepts 


4 3.72 


1.0 


4 


3.77 


1.0 


.083 


Ability to develop specialized scientific vocab 


4 3.58 


0.9 


4 


3.70 


0.9 


.023 


Ability to develop essay writing skills in 


3 3.40 


1.1 


4 


3.49 


1.1 


.024 


science 














Ability to develop scientific writing skills 


3 2.87 


1.3 


3 


2.88 


1.3 


.322 
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Academic Language Functions (Question 30) 



Indicate how often you used the following 
academic language functions; 


ELL students 




Non-ELL students 


f-test 

(p-val) 


Median 


Mean 


SD 


Median 


Mean 


SD 


Prediction of scientific outcomes 


4 


4.11 


1.0 


4 


4.13 


1.0 


.322 


Explanation of science processes 


4 


4.06 


1.0 


4 


4.11 


1.0 


.083 


Sequencing or organizing steps 


4 


4.03 


1.0 


4 


4.08 


1.0 


.159 


Definition of scientific vocabulary 


4 


3.91 


1.0 


4 


3.91 


1.0 


1.000 


Description of objects or phenomena 


4 


3.85 


1.1 


4 


3.89 


1.1 


.159 


Inference making based on science 


4 


3.79 


1.0 


4 


3.89 


0.9 


.058 


Labeling of scientific processes/objects 


4 


3.75 


1.0 


4 


3.79 


1.0 


.159 


Hypothesis generating based on science 


4 


3.75 


1.0 


4 


3.81 


1.0 


.083 


Sequencing or organizing information 


4 


3.60 


1.1 


4 


3.66 


1.1 


.083 


Causal reasoning with science 


4 


3.47 


1.0 


4 


3.51 


1.0 


.159 


Comparing/Contrasting processes/objects 


3 


3.36 


1.1 


4 


3.42 


1.1 


.083 


Generalization of scientific facts to other 


3 


3.32 


1.1 


3 


3.32 


1.1 


NA 


Enumeration of science facts/processes 


3 


3.26 


1.2 


3 


3.28 


1.2 


.569 


Classifying scientific phenomena 


3 


2.98 


1.1 


3 


3.00 


1.0 


.322 
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Appendix E 



Instructional Activities & Configurations (Question 17) 



Factor 



Item/Indicator 


Traditional 

Instmction 


Reform 
Instmction / 
Technology 


Reform 
Instmction / 
Language 
Modes 


Groupwork 


Listen to lecture/teacher instruction 


0.48 


0.03 


-0.02 


0.13 


Read a science textbook 


0.03 


-0.03 


0.80 


-0.14 


Read science articles in magazines/newspapers 


0.19 


0.42 


0.07 


0.01 


Complete worksheets 


0.48 


-0.17 


0.28 


0.12 


Conduct hands-on activities/manipulate 


0.35 


-0.10 


-0.17 


0.75 


Discuss science 


-0.23 


0.60 


0.09 


0.27 


Homework 


0.68 


-0.05 


-0.03 


0.00 


Watch videos 


0.08 


0.68 


-0.08 


-0.11 


Present oral science reports 


0.08 


0.20 


0.61 


0.11 


Complete a written science report 


-0.09 


0.07 


0.64 


0.13 


Use science-related software 


-0.04 


0.46 


0.02 


0.06 


Use science-related internet resources 


0.07 


0.33 


0.08 


-0.10 


Create graphic representations 


-0.07 


-0.03 


0.09 


0.78 


Work on individual projects 


0.35 


0.27 


0.11 


0.07 


Work on projects in pairs or groups 


-0.09 


0.26 


0.23 


0.56 


Work in groups arranged by ability 


0.52 


0.16 


-0.14 


0.00 


Work one-on-one with a teacher or an aide 


0.51 


0.07 


0.08 


-0.16 



Factor Intercorrelations 





FI 


F2 


F3 


F4 


FI 


1.00 








F2 


0.18 


1.00 






F3 


0.11 


0.33 


1.00 




F4 


0.08 


0.11 


0.22 


1.00 
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Instructional Emphases (Question 18) 



Factor 






Rote Learning / 


Science Contents 


Item/Indicator 


Connections 


Memory 


/ Concepts 


Learning about the relevance of science 


0.36 


0.45 


0.21 


Knowing basic science facts/terminology 


-0.25 


0.32 


0.96 


Understanding key science concepts 


0.12 


0.05 


0.72 


Developing test-taking skills 


0.02 


0.41 


0.08 


Memorization of previous test questions 


0.07 


0.64 


0.01 


Learning how science applies to everyday life 


0.75 


0.29 


0.01 


Using concepts to draw connections 


0.67 


0.35 


0.04 


Building on background scientific knowledge 


0.77 


-0.11 


0.03 


Developing laboratory skills or techniques 


0.36 


-0.24 


0.47 


Conducting inquiries or investigations 


0.36 


-0.48 


0.61 


Developing children’s interest in science 


0.75 


-0.08 


-0.06 


Using technology as a scientific tool 


0.28 


0.07 


0.18 



Factor Intercorrelations 

FI F2 F3 
FI 1.00 

F2 0.10 1.00 

F3 0.50 0.19 1.00 



Student Assessment (Question 19) 



Factor 


Item/Indicator 


Assessment 


Teacher made quizzes 


0.49 


Chapter (unit) end tests 


0.43 


Other assessments 


0.43 


Student work 


0.77 


Peer assessment 


0.82 


Student self-assessment 


0.68 
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Focus of Student Evaluation (Question 20) 



Item/Indicator 


Factor 

Behavioral Content 


Achievement/progress relative to class 


0.15 


0.41 


Achievement/progress relative to state 


0.30 


0.21 


Achievement/progress relative to prior ability 


0.79 


-0.11 


Effort 


0.83 


0.00 


Class participation 


0.83 


0.11 


Behavior/conduct 


0.66 


0.16 


Using scientific vocabulary 


0.10 


0.70 


Knowing scientific facts 


-0.18 


1.06 


Understanding of scientific concepts 


0.26 


0.51 



Factor Intercorrelations 





FI 


F2 


FI 


1.00 




F2 


0.61 


1.00 



Uses of Test Scores (Question 22) 



Factor 


Item/Indicator 


Assessment 


Assess student achievement and progress 


0.81 


Adapt/guide instruction 


0.80 


Provide feedback to students 


0.85 


Identify areas of strengths and weaknesses 


0.85 


Provide information for parents 


0.80 


Assign additional or different work in class 


0.93 


Assign additional or different homework 


0.92 
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ELL Instructional Practices (Question 27) 



Item/Indicator 


Teacher 

Strategies 


Eactor 

Student Print 

Strategies Materials 


Oral 

Language 


Scaffolding techniques to support English 


0.66 


0.11 


0.13 


-0.11 


Opportunities for student/teacher interactions 


0.32 


0.48 


-0.03 


-0.12 


Opportunities for student/student interactions 


0.10 


0.31 


0.03 


0.28 


Opportunities to practice English 


0.40 


0.06 


0.07 


0.12 


Opportunities to clarify vocabulary 


0.26 


-0.05 


0.68 


0.31 


Opportunities for peer assessment of English 


0.24 


0.05 


0.25 


0.65 


Opportunities for self assessment of English 


0.2 


0.01 


0.28 


0.75 


Supplementary ELL materials 


0.14 


0.52 


0.20 


0.27 


Adapt speech to all levels of proficiency 


0.52 


0.32 


0.25 


-0.06 


Adapt texts to all levels of proficiency 


0.32 


0.63 


-0.18 


-0.03 


Use tests with low levels of language demand 


-0.13 


0.89 


0.13 


-0.14 


Provide feedback about students English 


0.10 


0.40 


0.03 


0.22 


Use “English Only” 


-0.13 


0.06 


-0.39 


0.49 


Use simultaneous translation 


-0.24 


0.20 


0.57 


-0.03 


Incorporate students culture and background 


-0.08 


0.56 


0.06 


0.32 


Allow primary language in questions 


0.25 


-0.11 


0.76 


0.02 


Use primary language to clarify queries 


0.03 


0.05 


0.86 


0.07 


Use paraphrasing 


0.74 


0.06 


-0.02 


-0.01 


Use wait time to allow students to respond 


0.76 


-0.05 


-0.07 


0.18 



Factor Intercorrelations 





FI 


F2 


F3 


F4 


FI 


1.00 








F2 


0.23 


1.00 






F3 


0.28 


0.09 


1.00 




F4 


0.35 


0.25 


0.27 


1.00 
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ALE Instructional Strategies (Question 28) 



Item/Indicator 


Factor 

Strategies 


Links between new concepts and background 


0.66 


Scaffolding techniques for responses 


0.90 


Scaffolding techniques for expectations 


0.69 


Opportunities for students to clarify concepts 


0.76 


Opportunities to witness authentic discourse 


0.54 


Explicit language objectives 


0.73 


Opportunities to practice academic language 


0.64 



ALE Instructional Emphases (Question 29) 



Factor 



Item/Indicator 


General 

Language 


Specialized 

Language 


Ability to develop general academic vocab 


0.48 


0.33 


Ability to develop specialized scientific vocab 


-0.03 


0.75 


Ability to develop grammatical structures 


0.79 


-0.03 


Ability to develop active listening skills 


0.77 


0.05 


Ability to develop reading comprehension skills 


0.87 


-0.07 


Ability to develop scientific writing skills 


0.35 


0.28 


Ability to develop essay writing skills in science 


0.78 


-0.01 


Ability to convey basic scientific concepts 


-0.07 


0.92 


Ability to use evidence to justify science 


0.04 


0.71 


Ability to draw connections among others ideas 


0.26 


0.50 



Factor Intercorrelations 





FI 


F2 


FI 


1.00 




F2 


0.66 


1.00 
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Academic Language Functions (Question 30) 



Item/Indicator 




Factor 




Providing 

Information 


Higher Order 
Thinking 


Sorting / 
Organizing 


Explanation of science processes 


0.39 


0.34 


0.17 


Description of objects or phenomena 


0.81 


-O.II 


0.20 


Definition of scientific vocabulary 


0.46 


0.37 


0.03 


Classifying scientific phenomena 


O.IO 


-0.07 


0.89 


Comparing/Contrasting processes/objects 


0.13 


0.17 


0.63 


Causal reasoning with science 


0.26 


0.29 


0.37 


Inference making based on science 


0.08 


0.27 


0.54 


Sequencing or organizing steps 


0.88 


0.06 


-0.10 


Labeling of scientific processes/objects 


0.80 


-0.02 


0.13 


Enumeration of science facts/processes 


O.II 


0.36 


0.29 


Prediction of scientific outcomes 


0.28 


0.76 


-0.25 


Generalization of scientific facts to other 


0.28 


0.27 


0.28 


Sequencing or organizing information 


-0.04 


0.78 


0.21 


Hypothesis generating based on science 


-0.09 


0.81 


0.17 



Factor Intercorrelations 





FI 


F2 


F3 


FI 


1.00 






F2 


0.54 


1.00 




F3 


0.51 


0.63 


1.00 
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Appendix F: 

Correspondence Between Teacher Survey Responses and Classroom Observations 



SECTION IV: GENERAL INSTRUCTIONAL PRACTICES (OTL) 
Instructional Activities & Configurations (Question 17) 



Approximately how frequently 


Average Rating 


Agreement 


do the students in your science 


















class do the following? 


Classroom 


Teacher Self- 


Inter-Rater 


Observation- 


(Item was Recoded to 0-1 Scale) 


Observers 


Report 


(Observation) 


Survey 


Listen to a lecture/ Teacher-directed 
instruction 


1.0 


0.8 


1.00 


0.80 


Read a science textbook 


0.0 


1.0 


1.00 


0.00 


Read science articles in magazines or 
newspapers 


0.0 


0.4 


1.00 


0.60 


Complete worksheets 


0.6 


0.8 


0.90 


0.80 


Conduct hands-on activities or 
manipulate materials 


0.8 


1.0 


1.00 


1.00 


Discuss science 


0.7 


0.8 


0.90 


0.60 


Homework 


0.2 


0.6 


1.00 


0.60 


Watch videos 


0.0 


0.0 


1.00 


1.00 


Present oral science reports 


0.0 


0.0 


1.00 


1.00 


Complete written science reports 


0.0 


0.0 


1.00 


1.00 


Use science-related software 


0.0 


0.0 


1.00 


1.00 


Use science-related internet resources 


0.0 


0.0 


1.00 


1.00 


Create graphic representations 


0.5 


0.6 


0.80 


0.40 


Work on individual projects 


0.5 


0.5 


0.90 


0.50 


Work on projects in pairs or groups 


0.7 


1.0 


0.90 


1.00 


Work in groups arranged hy ability 


0.0 


0.6 


1.00 


0.40 


Work one-on-one with a teacher or an 
aide 


0.0 


0.4 


1.00 


0.60 


Section Total: 


0.29 


0.50 


0.96 


0.72 
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Instructional Emphasis (Question 18) 





Average Rating 


Agreement 


To what extent do your instructional 


Classroom 


Teacher Self- 


Inter-Rater 


Observation- 


practices emphasize the following? 


Observers 


Report 


(Observation) 


Survey 


Relevance to society 


1.0 


3.6 


0.90 


0.00 


Knowing basic facts 


3.0 


4.0 


1.00 


0.80 


Understanding key concepts 


3.2 


4.0 


0.75 


0.80 


Develop test taking skills 


0.0 


2.6 


1.00 


0.20 


Memorize previous test questions 


0.0 


1.6 


1.00 


0.20 


Learn how it applies to everyday life 


2.0 


3.8 


1.00 


0.60 


Draw connections between different 
concepts 


1.1 


3.6 


0.90 


0.00 


Build on background knowledge 


2.8 


3.8 


0.70 


0.60 


Develop lab skills 


1.4 


3.6 


0.70 


0.20 


Conduct inquiries 


2.4 


3.8 


0.70 


0.80 


Develop children's interest 


3.3 


4.0 


0.70 


0.40 


Use technology as a science tool 


0.0 


2.4 


1.00 


0.20 


Section Total: 


1.69 


3.40 


0.86 


0.40 
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SECTION V ; ELL SPECIFIC PRACTICES (OTL) 



ELL Instructional Practices (Question 27) 



To what extent do you rely on the 
following English language learner (ELL) 
instructional strategies in your classroom? 


Average Rating 


Agreement 


Classroom 

Observers 


Teacher Self- 
Report 


Inter-Rater 

(Observation) 


Observation- 

Survey 


Scaffolding techniques 


2.5 


4.6 


0.70 


0.00 


Student to teacher interactions 


3.4 


4.4 


1.00 


0.60 


Student to student interactions 


3.2 


4.4 


0.60 


0.40 


Opportunities to practice English 


2.9 


4.4 


0.90 


0.20 


Opportunities to clarify in primary 
language 


0.4 


4.2 


1.00 


0.00 


Opportunities for peer assessment 


0.6 


4.0 


0.70 


0.00 


Opportunities for self assessment 


0.2 


4.0 


1.00 


0.00 


Use supplementary materials 


0.0 


4.0 


1.00 


0.00 


Adapt speech 


0.9 


4.2 


0.80 


0.00 


Adapt texts 


0.4 


3.2 


0.80 


0.20 


Provide feedback 


1.3 


3.0 


0.70 


0.60 


Use "English Only" 


4.2 


1.8 


0.80 


0.00 


Use simultaneous translation 


0.1 


2.8 


1.00 


0.25 


Link to home culture 


1.0 


3.8 


0.78 


0.00 


Allow students to use primary language 


0.3 


2.2 


1.00 


0.20 


Use primary language to clarify queries 


0.3 


2.4 


1.00 


0.40 


Use paraphrasing 


2.3 


4.0 


0.60 


0.20 


Use wait time 


2.4 


4.6 


0.80 


0.00 


Section Total: 


1.45 


3.66 


0.84 


0.17 
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SECTION VI ; ACADEMIC LANGUAGE PRACTICES 



ALE Instructional Strategies (Question 28) 





Average Rating 


Agreement 


To what extent do you rely on the following 


Classroom 


Teacher 


Inter-Rater 


Observation 


academic English instructional strategies? 


Observers 


Self-Report 


(Observation) 


-Survey 


Explicit links to past learning 


2.9 


4.2 


0.60 


0.40 


Scaffolding student responses 


3.2 


4.2 


0.89 


0.60 


Scaffolding teacher expectations 


3.2 


3.8 


0.88 


0.60 


Providing opportunities to clarify concepts 


2.8 


4.2 


1.00 


0.40 


Providing opportunities to witness discourse 


0.0 


3.0 


1.00 


0.00 


Explicit language objectives 


2.3 


2.8 


0.78 


0.80 


Providing opportunities to practice English 


3.1 


4.0 


0.70 


0.40 


Section Total: 


2.47 


3.74 


0.84 


0.46 


ALE Instructional Emphases (Question 29) 




Average Rating 


Agreement 


To what extent do your instructional 


Classroom 


Teacher 


Inter-Rater 


Observation- 


practices emphasize the following? 


Observers 


Self-Report 


(Observation) 


Survey 


Academic vocabulary 


2.4 


4.2 


0.80 


0.20 


Scientific vocabulary 


3.1 


3.8 


0.89 


0.80 


Grammatical structures 


2.0 


3.6 


1.00 


0.20 


Reading comprehension 


2.5 


4.0 


0.70 


0.40 


Active listening 


2.2 


3.8 


0.90 


0.40 


Scientific writing skills 


1.7 


3.8 


0.63 


0.20 


Essay writing 


0.2 


3.6 


0.90 


0.00 


Convey concepts accurately and effectively 


2.5 


4.0 


0.89 


0.60 


Use evidence to explain 


2.6 


4.2 


1.00 


0.40 


Draw connections among student ideas 


2.5 


3.6 


0.90 


0.40 


Section Total: 


2.15 


3.86 


0.86 


0.36 
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Academic Language Functions (Question 30) 





Average 


Rating 


Agreement 


How often did you use the 


Classroom 


Teacher 


Inter-Rater 


Observation 


following academic language functions? 


Observers 


Self-Report 


(Observation) 


-Survey 


Explanation of science processes 


1.9 


3.8 


0.80 


0.00 


Description of scientific objects 


2.9 


3.8 


0.60 


0.60 


Definition of scientific vocabulary 


1.6 


4.0 


0.88 


0.20 


Classifying scientific phenomenon 


0.6 


3.4 


0.80 


0.20 


Comparing/contrasting scientific processes 


1.8 


3.4 


0.80 


0.40 


Causal reasoning with scientific phenomenon 


1.5 


3.6 


0.80 


0.20 


Sequencing or organizing extended information 


1.5 


4.0 


0.90 


0.00 


Sequencing steps of scientific procedures 


2.5 


4.0 


0.90 


0.40 


Labeling scientific processes 


0.4 


4.0 


0.90 


0.00 


Enumeration of science facts 


2.3 


4.0 


0.88 


0.20 


Prediction of scientific outcomes 


1.7 


4.2 


0.90 


0.20 


Generalizing to other scientific phenomenon 


0.5 


4.0 


0.78 


0.00 


Inference making based on scientific 
knowledge 


1.7 


3.8 


0.90 


0.20 


Hypothesis-generating based on scientific facts 


2.1 


4.0 


0.90 


0.20 


Section Total: 


1.63 


3.86 


0.84 


0.20 
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Appendix G: 

Patterns of Item inter-correlation by Scale 



Intercorrelations For Question 17: Traditional 





Lecture 


Worksheets 


Homework 


Individual 


Ability 


Aide 


Lecture 


1.0000 












Worksheets 


0.1663 

(0.2387) 


1.0000 










Homework 


0.2924 

(0.0354) 


0.4214 

(0.0017) 


1.0000 








Individual 


0.2488 

(0.0784) 


0.1492 

(0.2911) 


0.3752 

(0.0061) 


1.0000 






Ability 


0.2954 

(0.0335) 


0.2567 

(0.0635) 


0.3404 

(0.0126) 


0.1285 

(0.3640) 


1.0000 




Aide 


0.3415 

(0.0142) 


0.2191 

(0.1186) 


0.2229 

(0.1123) 


0.2831 

(0.0441) 


0.2623 

(0.0603) 


1.0000 


Intercorrelations For Question 17: Reform/Technology 








Articles 


Discuss 


Videos 


Software 


Internet 




Articles 


1.0000 












Discuss 


0.3411 

(0.0124) 


1.0000 










Videos 


0.3396 

(0.0129) 


0.3056 

(0.0260) 


1.0000 








Software 


0.0479 

(0.7332) 


0.3145 

(0.0218) 


0.3833 

(0.0046) 


1.0000 






Internet 


0.0291 

(0.8362) 


0.3037 

(0.0271) 


0.2122 

(0.1271) 


0.2336 

(0.0923) 


1.0000 
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Intercorrelations For Question 17: Reform/Language Modes 





Textbook 


Present 


Write 


Textbook 


1.0000 






Present 


0.4565 

(0.0007) 


1.0000 




Write 


0.4969 

(0.0002) 


0.5865 

(0.0000) 


1.0000 


Intercorrelations For Question 17: 


Hands-on & Group 




Activities 


Graphics 


Pairs 


Activities 


1.0000 






Graphics 


0.5227 

(0.0001) 


1.0000 




Pairs 


0.4108 

(0.0023) 


0.4785 

(0.0003) 


1.0000 


Intercorrelations For Question 18: Making Connections 





Relevance 


Application 


Connections 


Background 


Interest 


Technology 


Relevance 


1.0000 












Application 


0.6959 

(0.0000) 


1.0000 










Connections 


0.4458 


0.6778 


1.0000 










(0.0008) 


(0.0000) 










Background 


0.3280 


0.5515 


0.5201 


1.0000 








(0.0165) 


(0.0000) 


(0.0001) 








Interest 


0.2760 


0.4828 


0.4732 


0.6242 


1.0000 






(0.0455) 


(0.0003) 


(0.0003) 


(0.0000) 






Technology 


0.2832 


0.2460 


0.3579 


0.3100 


0.3241 


1.0000 




(0.0399) 


(0.0758) 


(0.0085) 


(0.0239) 


(0.0179) 




Intercorrelations For Question 18: Rote Learning/Memorization 





Test Skills 


Memorization 


Test Skills 


1.0000 




Memorization 


0.4155 


1.0000 




(0.0020) 
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Intercorrelations For Question 18: Understanding Science Contents/Concepts 





Concepts 


Facts 


Labs 


Inquiry 


Concepts 


1.0000 








Facts 


0.7400 


1.0000 








(0.0000) 








Labs 


0.4282 


0.3988 


1.0000 






(0.0014) 


(0.0031) 






Inquiry 


0.5452 


0.3490 


0.7518 


1.0000 




(0.0000) 


(0.0104) 


(0.0000) 




Intercorrelations For Question 19 





Quizzes 


Tests 


Peer 


Self 


Work 


Other 


Quizzes 


1.0000 












Tests 


0.2398 


1.0000 












(0.0837) 












Peer 


0.3953 


0.3210 


1.0000 










(0.0034) 


(0.0191) 










Self 


0.2177 


0.3970 


0.7046 


1.0000 








(0.1173) 


(0.0032) 


(0.0000) 








Work 


0.3550 


0.3263 


0.6207 


0.4730 


1.0000 






(0.0091) 


(0.0171) 


(0.0000) 


(0.0003) 






Other 


0.2857 


0.1063 


0.2026 


0.2772 


0.3898 


1.0000 




(0.0400) 


(0.4534) 


(0.1498) 


(0.0467) 


(0.0043) 





Intercorrelations For Question 20: Behavioral 




Previous 


Effort Participation Behavior 


Previous 


1.0000 




Effort 


0.5842 

(0.0000) 


1.0000 


Participation 


0.6072 

(0.0000) 


0.8017 1.0000 

(0.0000) 


Behavior 


0.5877 

(0.0000) 


0.6302 0.6678 1.0000 

(0.0000) (0.0000) 
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Intercorrelations For Question 20: Content 





Vocabulary 


Facts 


Concepts 


To Class 


To State 




Vocabulary 


1.0000 














Facts 


0.7062 

(0.0000) 


1.0000 












Concepts 


0.5404 

(0.0000) 


0.6637 

(0.0000) 


1.0000 










To Class 


0.3855 

(0.0044) 


0.4494 

(0.0007) 


0.2300 

(0.0975) 


1.0000 






To State 


0.2582 

(0.0646) 


0.2770 

(0.0468) 


0.3558 

(0.0096) 


0.4763 

(0.0004) 


1.0000 




Intercorrelations For Question 22 














Achievement 


Instruction 


Feedback 


Strengths 


Parents 


Classwork 


Homework 


Achievement 


1.0000 














Instruction 


0.7611 

(0.0000) 


1.0000 












Feedback 


0.7351 

(0.0000) 


0.7417 

(0.0000) 


1.0000 










Strengths 


0.6835 

(0.0000) 


0.6455 

(0.0000) 


0.6912 

(0.0000) 


1.0000 








Parents 


0.6192 

(0.0000) 


0.6874 

(0.0000) 


0.6485 

(0.0000) 


0.6810 

(0.0000) 


1.0000 






Classwork 


0.6918 

(0.0000) 


0.6395 

(0.0000) 


0.7667 

(0.0000) 


0.8056 

(0.0000) 


0.7482 

(0.0000) 


1.0000 




Homework 


0.6728 

(0.0000) 


0.6714 

(0.0000) 


0.7451 

(0.0000) 


0.8040 

(0.0000) 


0.7388 

(0.0000) 


0.9655 

(0.0000) 


1.0000 
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Intercorrelations For Question 27: Support Content Learning 





Scaffolding 


StudentTeacher 


Paraphrasing 


Wait Time 


Feedback 


Scaffolding 


1.0000 










StudentTeacher 


0.5019 


1.0000 










(0.0001) 










Paraphrasing 


0.4853 


0.4576 


1.0000 








(0.0002) 


(0.0006) 








Wait Time 


0.5257 


0.2996 


0.6375 


1.0000 






(0.0001) 


(0.0293) 


(0.0000) 






Feedback 


0.3506 


0.4149 


0.2906 


0.2646 


1.0000 




(0.0101) 


(0.0020) 


(0.0348) 


(0.0556) 





Intercorrelations For Question 27: Student Strategies 




StudentStudent 


Peer Assess 


Self Assess 


StudentStudent 


1.0000 






Peer Assess 


0.2112 


1.0000 






(0.1289) 






Self Assess 


0.3105 


0.8748 


1.0000 




(0.0237) 


(0.0000) 





Intercorrelations For Question 27: Oral Language Strategies 





Clarify 


Adapt 

Speech 


English 

Only 


Translation 


Allow 

Primary 


Use 

Primary Practice 


Clarify 


1.0000 
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Intercorrelations For Question 27: Materials/Background 
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Intercorrelations For Question 28 
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Intercorrelations For Question 29: General Language Skills 
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Intercorrelations For Question 29: Specialized Language 
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Intercorrelations For Question 30: Sorting/Organizing Information 
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Intercorrelations For Question 30: Providing Information 
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Intercorrelations For Question 30: Higher Order Thinking 
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Appendix H: 

Additional High-Contrast Case Studies 
Science Instruction in Mr. Grahame’s Classroom (Low # of ELL students) 

Background information. Mr. Grahame has been teaching for 16 years which is 
double the mean for the survey sample. He has a Bachelor’s degree in Mathematics and 
holds both an elementary teaching credential and the Crosscultural, Language and Academic 
Development (CLAD) certificate from the State of California, not surprisingly then, Mr. 
Grahame is proficient in both Spanish and English. He has been teaching science at the 4th 
grade for seven years which puts him over the average of just over four for the survey 
sample. While Mr. Grahame reports taking 10-12 course for general pedagogy and teaching 
methods, he has also taken fewer specialized training courses (i.e.. Science and ELL 
methods) than the sample overall, with just one-two courses of each. 

There are 31 students in Mr. Grahame’s classroom, six more students than the survey 
average. He reports almost equal numbers of boys and girls. The majority of students (87%) 
are at the “Basic” level on the CST for LEA. A further 13% are below Basic or far below 
Basic, and just 9% are at or above the “Proficient” levels. Mr. Grahame was not sure whether 
any of his students came from economically disadvantaged backgrounds, but given he 
teaches at the same school as Mrs. Troy we can assume that the demographics are 
comparable in the two classrooms. Lour (13%) students were reported to be ELL students at 
the “advanced beginner” level according to Mr. Grahame. To accommodate these students’ 
learning he reports using lots of manipulatives and speaks Spanish to the students. In terms of 
science instruction, the class receives three 60-minute lessons per week. Three observers 
were involved in rating these lessons; one observer was present at both sessions but joined by 
a different observer each time. In his debriefing interview, Mr. Grahame reported that the 
two lessons we observed were mostly typical of his instruction (other occasions he has 
included reading followed by discussion) 

Summary. Overall, the observer ratings bare close resemblance to those of Mr. 
Grahame’s reported practices on the survey. With the exception of developing laboratory 
techniques rated higher by one pair of observers, the teacher survey ratings were almost 
always slightly higher than the observer ratings. In terms of Opportunity for Science 
Learning and consistent with our observations, Mr. Grahame reported a relatively wide 
repertoire of science instructional practices and a mid to high emphasis on all instructional 
practices with the exception of a lower emphasis on rote memorization. Unlike most of the 
survey respondents, Mr. Grahame reported different levels of emphases for ELL and Non- 
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ELL students: He placed greater emphasis on knowing basic facts and understanding key 
concepts and how they applied to everyday life for ELL students than for Non-ELL students. 
In terms of ELL-Specific Science Instruction, contrary to our observations, Mr. Grahame 
reported frequently using a multitude of ELL instructional strategies and no “English-only” 
approaches. 

There was, however, slightly better agreement between Mr. Grahame’ s report and our 
observations for Academic Language Exposure, particularly with the wide variety and the 
mid to high emphases on academic language practices observed at the second session if not 
at the first session. Again, Mr. Grahame reported spending different amounts of time on 
certain language functions for ELL and Non-ELL students with slightly less time allotted to 
most functions with ELLs. While he reported mid to high values for nearly all functions 
across the two groups of students, the observers reported a focus on a narrower repertoire 
much less frequently used (i.e., description, definition, sequencing procedural steps, labeling, 
enumeration, and at the first session hypothesis -generation). Linally, in his debriefing 
interview, Mr. Grahame gave feedback on the survey itself, indicating that he thought the 
content was “95% very applicable” of his instructional practices in science. 

Science Instruction in Ms. Llosa’s Classroom (High # of ELL students) 

Background information. With just three years of teaching, Ms. Llosa was the most 
junior of the case study teachers we observed and thee current school year was the first year 
she had taught science at the 4th grade. She has a Bachelor’s degree in Child Development 
and holds both elementary and bilingual teaching credentials. Ms. Llosa is proficient in both 
Spanish and English. While she reported no specific in-service courses she ahs taken she 
mentioned weekly planning sessions as the source for training in science content and 
teaching methods as well as ELL teaching methods and general pedagogy and teaching 
methods. 

The composition of Mrs. Llosa’s classroom mirrors that of her colleague Ms. Gomez, 
with 25 students and the 17 girls far exceeding the eight boys. All students are ELL students 
from disadvantaged backgrounds. Ms. Llosa reported using Spanish that the students already 
knew in order to make sure they understood science concepts. This teacher also did not report 
the percentage of students at the different levels of the CST for ELA. Ms. Llosa reports 
teaching two-four science lessons per week that, at just 35-45 minutes each, last below the 
sample average of 49 minutes. Three observers were involved in rating these lessons; one 
observer was present at both sessions but joined by a different observer each time. In her 
debriefing interview, Ms. Llosa reported that the two lessons we observed were “pretty” 
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typical of her instruction with the format of a “mini-lesson” followed by an activity. Other 
formats she uses that were not observed in these lessons, included research on computers, 
read-alouds and watching videos. 

Summary. Overall, the observer ratings and those of Ms. Llosa’s reported practices on 
the survey were very similar. In terms of Opportunity for Science Learning, while the 
observers rated a subset of the instructional practices (e.g., knowing basic facts, conducting 
inquiries) reported by Ms. Llosa, the degree of emphases for these was comparably high in 
many cases. In terms of ELL-Specific Science Instruction, Ms. Llosa reported using all the 
listed strategies nearly always with the exception of “English-only” approaches which she 
rated as 2. The observers also reported observing a large number of strategies but there was 
some discrepancy across observers in the values of the ratings (ranging from low to high), 
although they did agree with each other and with Ms. Llosa on the non-exclusivity of an 
“English-only” approach. 

With the notable exceptions of not observing opportunities for authentic scientific 
discourse or any emphasis on science essay writing skills, there was generally good 
agreement between our observations of Academic Language Exposure and Ms. Llosa’s 
reported use of academic English instructional strategies and emphases. She had a wide 
repertoire here and this was self-rated as mid to high and largely corroborated by the 
observers. Ms. Llosa gave a high rating to the use of nearly all academic language functions. 
The observers reported a slightly narrower repertoire, somewhat less frequently observed 
(i.e., at the first session, definition, sequencing procedural steps, prediction and hypothesis- 
generation, and at the second session, comparing/contesting, sequencing information, 
sequencing procedural steps, enumeration, prediction, inference, and hypothesis-generation). 
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